{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practice XGBoost Parameter Tuning on Rental Listing Inquiries Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据说明\n",
    "Rental Listing Inquiries 数据集是 Kaggle 平台上的一个分类竞赛任务，需要根据公寓的特征来预测其受欢迎程度(用户感兴趣程度分为高、中、低三类)。  \n",
    "其中房屋的特征 x 共有 14 维，响应值 y 为用户对该公寓的感兴趣程度。  \n",
    "评价标准为 logloss。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 要求\n",
    "独立调用xgboost或在sklearn框架下调用均可。  \n",
    "1. 模型训练：超参数调优  \n",
    "a) 初步确定弱学习器数目  \n",
    "b) 对树的最大深度（可选）和min_children_weight进行调优（可选）  \n",
    "c) 对正则参数进行调优  \n",
    "d) 重新调整弱学习器数目  \n",
    "e) 行列重采样参数调整  \n",
    "2. 调用模型进行测试  \n",
    "3. 生成测试结果文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 导入必要的模块"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "# 导入 XGBoost 和 XGBoost 分类器\n",
    "import xgboost as xgb\n",
    "from xgboost import XGBClassifier\n",
    "# 导入模型选择交叉验证模块\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "# 导入评价指标计算模块 logloss\n",
    "from sklearn.metrics import log_loss\n",
    "# 导入绘图模块\n",
    "from matplotlib import pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dpath = './data/'\n",
    "dtrain = pd.read_csv(dpath + 'RentListingInquries_FE_train.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(49352, 228)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看数据集维度\n",
    "dtrain.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "共计 49352 个样本."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>price</th>\n",
       "      <th>price_bathrooms</th>\n",
       "      <th>price_bedrooms</th>\n",
       "      <th>room_diff</th>\n",
       "      <th>room_num</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>...</th>\n",
       "      <th>walk</th>\n",
       "      <th>walls</th>\n",
       "      <th>war</th>\n",
       "      <th>washer</th>\n",
       "      <th>water</th>\n",
       "      <th>wheelchair</th>\n",
       "      <th>wifi</th>\n",
       "      <th>windows</th>\n",
       "      <th>work</th>\n",
       "      <th>interest_level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.5</td>\n",
       "      <td>3</td>\n",
       "      <td>3000</td>\n",
       "      <td>1200.0</td>\n",
       "      <td>750.000000</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>4.5</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>5465</td>\n",
       "      <td>2732.5</td>\n",
       "      <td>1821.666667</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2850</td>\n",
       "      <td>1425.0</td>\n",
       "      <td>1425.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3275</td>\n",
       "      <td>1637.5</td>\n",
       "      <td>1637.500000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>18</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3350</td>\n",
       "      <td>1675.0</td>\n",
       "      <td>670.000000</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>28</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 228 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   bathrooms  bedrooms  price  price_bathrooms  price_bedrooms  room_diff  \\\n",
       "0        1.5         3   3000           1200.0      750.000000       -1.5   \n",
       "1        1.0         2   5465           2732.5     1821.666667       -1.0   \n",
       "2        1.0         1   2850           1425.0     1425.000000        0.0   \n",
       "3        1.0         1   3275           1637.5     1637.500000        0.0   \n",
       "4        1.0         4   3350           1675.0      670.000000       -3.0   \n",
       "\n",
       "   room_num  Year  Month  Day       ...        walk  walls  war  washer  \\\n",
       "0       4.5  2016      6   24       ...           0      0    0       0   \n",
       "1       3.0  2016      6   12       ...           0      0    0       0   \n",
       "2       2.0  2016      4   17       ...           0      0    0       0   \n",
       "3       2.0  2016      4   18       ...           0      0    0       0   \n",
       "4       5.0  2016      4   28       ...           0      0    1       0   \n",
       "\n",
       "   water  wheelchair  wifi  windows  work  interest_level  \n",
       "0      0           0     0        0     0               1  \n",
       "1      0           0     0        0     0               2  \n",
       "2      0           0     0        0     0               0  \n",
       "3      0           0     0        0     0               2  \n",
       "4      0           0     0        0     0               2  \n",
       "\n",
       "[5 rows x 228 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看数据集概貌\n",
    "dtrain.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "共 227 维特征(最后一列为分类标签)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# 数据准备\n",
    "y_train = dtrain['interest_level']\n",
    "dtrain = dtrain.drop(['interest_level'], axis=1)\n",
    "X_train = np.array(dtrain)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看 Target 分布"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Number of Occurences')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEKCAYAAADaa8itAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAGvlJREFUeJzt3Xu0HWWd5vHvQwBFRcPlyMQkGKTj\n2KBtgNOQNrNsQIWAjkFbFFaPRIc18QKKS9s2dDOk5dKtbQtLehQnNmnCLIcQ0JaIwXREkMGRkATD\nJVyGY0Q5koHQ4a6AwWf+qPcMOyf7nFOJtbPZyfNZq9au+tVbtX97nZX8VtX71luyTURERBN26XYC\nERGx40hRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGN2bXbCWxv\n++67r6dMmdLtNCIiesrq1asfsd03VrudrqhMmTKFVatWdTuNiIieIukXddrl9ldERDSmY0VF0ksl\n3SLpNklrJX2+xC+V9HNJa8oyrcQl6SJJA5Jul3Roy7lmS7qvLLNb4odJuqMcc5Ekder3RETE2Dp5\n++tZ4GjbT0naDbhJ0rVl32dtXzWs/XHA1LIcAVwMHCFpb2Ae0A8YWC1pie1HS5s5wM3AUmAmcC0R\nEdEVHbtSceWpsrlbWUabZ38WcFk57mZgvKQJwLHActsbSyFZDsws+15p+yeu5u+/DDihU78nIiLG\n1tE+FUnjJK0BHqYqDCvKrvPLLa4LJb2kxCYCD7QcPlhio8UH28Tb5TFH0ipJqzZs2PB7/66IiGiv\no0XF9vO2pwGTgMMlvRE4E3gD8MfA3sDnSvN2/SHehni7PObb7rfd39c35oi4iIjYRttl9Jftx4Ab\ngJm215dbXM8C/wwcXpoNApNbDpsEPDhGfFKbeEREdEknR3/1SRpf1vcA3g7cU/pCKCO1TgDuLIcs\nAU4po8CmA4/bXg8sA46RtJekvYBjgGVl35OSppdznQJc3anfExERY+vk6K8JwEJJ46iK12Lb10j6\noaQ+qttXa4CPlvZLgeOBAeDXwIcBbG+UdC6wsrQ7x/bGsv4x4FJgD6pRXxn5FRHRRaoGTu08+vv7\nnSfqI17cZvzjjG6nsMP78Sd+vFXtJa223T9WuzxRHxERjUlRiYiIxqSoREREY1JUIiKiMSkqERHR\nmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGpKhERERjUlQiIqIxKSoREdGYFJWIiGhMikpE\nRDQmRSUiIhqTohIREY1JUYmIiMakqERERGNSVCIiojEpKhER0ZiOFRVJL5V0i6TbJK2V9PkSP0DS\nCkn3SbpC0u4l/pKyPVD2T2k515klfq+kY1viM0tsQNLcTv2WiIiop5NXKs8CR9t+MzANmClpOvBF\n4ELbU4FHgVNL+1OBR23/AXBhaYekg4CTgIOBmcDXJI2TNA74KnAccBBwcmkbERFd0rGi4spTZXO3\nshg4GriqxBcCJ5T1WWWbsv9tklTii2w/a/vnwABweFkGbK+z/RywqLSNiIgu6WifSrmiWAM8DCwH\nfgY8ZntTaTIITCzrE4EHAMr+x4F9WuPDjhkpHhERXdLRomL7edvTgElUVxZ/2K5Z+dQI+7Y2vgVJ\ncyStkrRqw4YNYyceERHbZLuM/rL9GHADMB0YL2nXsmsS8GBZHwQmA5T9rwI2tsaHHTNSvN33z7fd\nb7u/r6+viZ8UERFtdHL0V5+k8WV9D+DtwN3A9cD7SrPZwNVlfUnZpuz/oW2X+ElldNgBwFTgFmAl\nMLWMJtudqjN/Sad+T0REjG3XsZtsswnAwjJKaxdgse1rJN0FLJJ0HvBT4JLS/hLgf0gaoLpCOQnA\n9lpJi4G7gE3AabafB5B0OrAMGAcssL22g78nIiLG0LGiYvt24JA28XVU/SvD488AJ45wrvOB89vE\nlwJLf+9kIyKiEXmiPiIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0\nJkUlIiIak6ISERGNSVGJiIjGpKhERERjxiwqkv5e0isl7SbpOkmPSPpP2yO5iIjoLXWuVI6x/QTw\nLqoXY70e+GxHs4qIiJ5Up6jsVj6PBy63vbGD+URERA+r8z6V70q6B/gN8HFJfcAznU0rIiJ60ZhX\nKrbnAn8C9Nv+LfBrYFanE4uIiN5Tp6P+ZcBpwMUl9Bqgv5NJRUREb6rTp/LPwHPAW8r2IHBexzKK\niIieVaeoHGj774HfAtj+DaCOZhURET2pTlF5TtIegAEkHQg829GsIiKiJ9UpKvOA7wOTJX0TuA74\ny7EOkjRZ0vWS7pa0VtIZJf43kn4laU1Zjm855kxJA5LulXRsS3xmiQ1ImtsSP0DSCkn3SbpC0u5b\n8dsjIqJhYw4ptr1c0q3AdKrbXmfYfqTGuTcBn7F9q6Q9gdWSlpd9F9r+h9bGkg4CTgIOphoM8ANJ\nry+7vwq8g6o/Z6WkJbbvAr5YzrVI0teBU3lhQEFERGxndUZ/vQfYZPt7tq8BNkk6YazjbK+3fWtZ\nfxK4G5g4yiGzgEW2n7X9c2AAOLwsA7bX2X4OWATMkiTgaOCqcvxCYMy8IiKic2rd/rL9+NCG7ceo\nbonVJmkKcAiwooROl3S7pAWS9iqxicADLYcNlthI8X2Ax2xvGhaPiIguqVNU2rWp8yQ+AJJeAXwL\n+FSZQ+xi4EBgGrAe+PJQ0zaHexvi7XKYI2mVpFUbNmyom3pERGylOkVllaQLJB0o6XWSLgRW1zm5\npN2oCso3bX8bwPZDtp+3/TvgG1S3t6C60pjccvgk4MFR4o8A4yXtOiy+Bdvzbffb7u/r66uTekRE\nbIM6ReUTVA8/XgFcSTXv12ljHVT6PC4B7rZ9QUt8Qkuz9wB3lvUlwEmSXiLpAGAqcAuwEphaRnrt\nTtWZv8S2geuB95XjZwNX1/g9ERHRIXVGfz0NzB2rXRszgA8Cd0haU2J/BZwsaRrVrar7gY+U71kr\naTFwF9XIsdNsPw8g6XRgGTAOWGB7bTnf54BFks4DfkpVxCIiokvGLCplWO9fAFNa29s+erTjbN9E\n+36PpaMccz5wfpv40nbH2V7HC7fPIiKiy+p0uF8JfB34J+D5zqYTERG9rE5R2WQ7DxRGRMSY6nTU\nf1fSxyVNkLT30NLxzCIioufUuVKZXT5b30tv4HXNpxMREb2szuivA7ZHIhER0ftqvflR0lmS5pft\nqZLe1fnUIiKi1+TNjxER0Zi8+TEiIhqTNz9GRERj6oz+Gv7mxxnAhzqZVERE9KZRi0qZFPIe4L1s\n/ZsfIyJiJzNqUbFtSd+xfRjwve2UU0RE9Kg6fSo3S/rjjmcSERE9r06fylHARyT9Ania6haYbf9R\nRzOLiIieU6eoHNfxLCIiYodQp6i0fe97RETEcHWKyveoCouAlwIHAPcCB3cwr4iI6EF1JpR8U+u2\npEMprwCOiIhoVWf012Zs3wpkNFhERGyhzjvqP92yuQtwKLChYxlFRETPqtOnsmfL+iaqPpZvdSad\niIjoZXX6VD6/LSeWNBm4DPh3wO+A+ba/Ul5FfAUwBbgfeL/tR8uUMF8Bjgd+DXyo3GpD0mzgrHLq\n82wvLPHDgEuBPYClVFPIZLRaRESX1HlJ13JJ41u295K0rMa5NwGfsf2HVPOGnSbpIGAucJ3tqcB1\nZRuq52GmlmUOcHH5vr2pJrU8AjgcmCdpr3LMxaXt0HEza+QVEREdUqejvs/2Y0Mbth8FXj3WQbbX\nD11p2H4SuBuYCMwCFpZmC4ETyvos4DJXbgbGS5oAHAsst72xfPdyYGbZ90rbPylXJ5e1nCsiIrqg\nTlF5XtL+QxuSXstWPhApaQpwCLAC2M/2eqgKDy8UqInAAy2HDZbYaPHBNvGIiOiSOh31fw3cJOlH\nZfutVLecapH0CqqO/U/ZfqLqOmnftE3M2xBvl8McSs77779/uyYREdGAMa9UbH+fahjxFcBi4DDb\ndfpUkLQbVUH5pu1vl/BD5dYV5fPhEh8EJrccPgl4cIz4pDbxdr9hvu1+2/19fX11Uo+IiG1Qp6P+\nPcBvbV9j+7vAJklj9l2U0VyXAHfbvqBl1xJgdlmfDVzdEj9FlenA4+X22DLgmDJAYC/gGGBZ2fek\npOnlu05pOVdERHRBnT6VebYfH9oonfbzahw3A/ggcLSkNWU5HvgC8A5J9wHvKNtQDQleBwwA3wA+\nXr5vI3AusLIs55QYwMeAfyrH/Ay4tkZeERHRIXX6VNoVnjrPt9xE+34PgLe1aW/gtBHOtQBY0Ca+\nCnjjWLlERMT2UedKZZWkCyQdKOl1ki4EVnc6sYiI6D11isongOeoOuqvBJ5hhCuKiIjYudW5jfW0\npPOAc20/vR1yioiIHjXqlYqkj0v6JfAL4JeSfiHp49sntYiI6DUjFhVJZwHvAo60vY/tfYCjgOPK\nvoiIiM2MdqXyQeC9ttcNBcr6+6meCYmIiNjMqLe/bD/TJvYbqqnsIyIiNjNaURmUtMXzJJKOBtZ3\nLqWIiOhVo43++iRwtaSbqJ5LMdW76WdQTVMfERGxmRGvVGyvpXpa/UaqtzS+rqy/seyLiIjYzKjP\nqZQ+lS2mR4mIiGinzhP1ERERtaSoREREY0Z7+PG68vnF7ZdORET0stH6VCZI+lPg3ZIWMWwae9u3\ndjSziIjoOaMVlbOBuVSv6b1g2D4DR3cqqYiI6E0jFhXbVwFXSfqvts/djjlFRESPqjP1/bmS3g28\ntYRusH1NZ9OKiIheNOboL0l/B5wB3FWWM0osIiJiM3XeUf9OYJrt3wFIWgj8FDizk4lFRETvqfuc\nyviW9Vd1IpGIiOh9da5U/g74qaTrqYYVv5VcpURERBtjXqnYvhyYDny7LH9ie9FYx0laIOlhSXe2\nxP5G0q8krSnL8S37zpQ0IOleSce2xGeW2ICkuS3xAyStkHSfpCsk7V7/Z0dERCfUuv1le73tJbav\ntv1/a577UmBmm/iFtqeVZSmApIOAk4CDyzFfkzRO0jjgq8BxwEHAyaUtwBfLuaYCjwKn1swrIiI6\npGNzf9m+EdhYs/ksYJHtZ23/HBgADi/LgO11tp8DFgGzJInq4curyvELgRMa/QEREbHVujGh5OmS\nbi+3x/YqsYnAAy1tBktspPg+wGO2Nw2LtyVpjqRVklZt2LChqd8RERHDjFpUJO3S2ifSgIuBA4Fp\nVK8k/vLQV7Vp622It2V7vu1+2/19fX1bl3FERNQ2alEpz6bcJmn/Jr7M9kO2ny/n/QbV7S2orjQm\ntzSdBDw4SvwRYLykXYfFIyKii+rc/poArJV0naQlQ8u2fJmkCS2b7wGGroKWACdJeomkA4CpwC3A\nSmBqGem1O1Vn/hLbBq4H3leOnw1cvS05RUREc+o8p/L5bTmxpMuBI4F9JQ0C84AjJU2julV1P/AR\nANtrJS2mmgZmE3Ca7efLeU4HlgHjgAW215av+BywSNJ5VE/4X7IteUZERHPqTCj5I0mvBaba/oGk\nl1H9Bz/WcSe3CY/4H7/t84Hz28SXAkvbxNfxwu2ziIh4EagzoeR/oRq6+99LaCLwnU4mFRERvalO\nn8ppwAzgCQDb9wGv7mRSERHRm+oUlWfLg4cAlBFXIw7fjYiInVedovIjSX8F7CHpHcCVwHc7m1ZE\nRPSiOkVlLrABuINqtNZS4KxOJhUREb2pzuiv35UXc62guu11b3lOJCIiYjNjFhVJ7wS+DvyManqU\nAyR9xPa1nU4uIiJ6S52HH78MHGV7AEDSgcD3gBSViIjYTJ0+lYeHCkqxDni4Q/lEREQPG/FKRdJ7\ny+paSUuBxVR9KidSzckVERGxmdFuf/3HlvWHgD8t6xuAvbZsHhERO7sRi4rtD2/PRCIiovfVGf11\nAPAJYEpre9vv7lxaERHRi+qM/voO1ezC3wV+19l0IiKil9UpKs/YvqjjmURERM+rU1S+Imke8K/A\ns0NB27d2LKuIiOhJdYrKm4APAkfzwu0vl+2IiIj/r05ReQ/wutbp7yNe7H55zpu6ncJOYf+z7+h2\nCvEiU+eJ+tuA8Z1OJCIiel+dK5X9gHskrWTzPpUMKY6IiM3UKSrzOp5FRETsEMa8/WX7R+2WsY6T\ntEDSw5LubIntLWm5pPvK514lLkkXSRqQdLukQ1uOmV3a3ydpdkv8MEl3lGMukqSt//kREdGkMYuK\npCclPVGWZyQ9L+mJGue+FJg5LDYXuM72VOC6sg1wHDC1LHOAi8t37011pXQEcDgwb6gQlTZzWo4b\n/l0REbGd1blS2dP2K8vyUuDPgP9W47gbgY3DwrOAhWV9IXBCS/wyV24GxkuaABwLLLe90fajwHJg\nZtn3Sts/KW+hvKzlXBER0SV1Rn9txvZ32PZnVPazvb6cZz3w6hKfCDzQ0m6wxEaLD7aJtyVpjqRV\nklZt2LBhG1OPiIix1JlQ8r0tm7sA/VQPPzapXX+ItyHelu35wHyA/v7+pnOPiIiizuiv1veqbALu\np7pdtS0ekjTB9vpyC2voDZKDwOSWdpOAB0v8yGHxG0p8Upv2ERHRRWMWlYbfq7IEmA18oXxe3RI/\nXdIiqk75x0vhWQb8bUvn/DHAmbY3lgEE04EVwCnAPzaYZ0REbIPRXid89ijH2fa5o51Y0uVUVxn7\nShqkGsX1BWCxpFOBX1K9mhhgKXA8MAD8Gvhw+ZKNks7lhdcXn2N7qPP/Y1QjzPYAri1LRER00WhX\nKk+3ib0cOBXYBxi1qNg+eYRdb2vT1sBpI5xnAbCgTXwV8MbRcoiIiO1rtNcJf3loXdKewBlUVxCL\ngC+PdFxEROy8Ru1TKQ8ffhr4c6rnSg4tz4tERERsYbQ+lS8B76Uaivsm209tt6wiIqInjfbw42eA\n1wBnAQ+2TNXyZM1pWiIiYiczWp/KVj9tHxERO7cUjoiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiI\nxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGpKhERERjUlQi\nIqIxKSoREdGYrhQVSfdLukPSGkmrSmxvScsl3Vc+9ypxSbpI0oCk2yUd2nKe2aX9fZJmd+O3RETE\nC7p5pXKU7Wm2+8v2XOA621OB68o2wHHA1LLMAS6GqggB84AjgMOBeUOFKCIiuuPFdPtrFrCwrC8E\nTmiJX+bKzcB4SROAY4HltjfafhRYDszc3klHRMQLulVUDPyrpNWS5pTYfrbXA5TPV5f4ROCBlmMH\nS2ykeEREdMmuXfreGbYflPRqYLmke0ZpqzYxjxLf8gRV4ZoDsP/++29trhERUVNXrlRsP1g+Hwb+\nhapP5KFyW4vy+XBpPghMbjl8EvDgKPF23zffdr/t/r6+viZ/SkREtNjuRUXSyyXtObQOHAPcCSwB\nhkZwzQauLutLgFPKKLDpwOPl9tgy4BhJe5UO+mNKLCIiuqQbt7/2A/5F0tD3/0/b35e0Elgs6VTg\nl8CJpf1S4HhgAPg18GEA2xslnQusLO3Osb1x+/2MiIgYbrsXFdvrgDe3if8b8LY2cQOnjXCuBcCC\npnOMiIht82IaUhwRET0uRSUiIhrTrSHFPeGwz17W7RR2eKu/dEq3U4iIBuVKJSIiGpOiEhERjUlR\niYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGpKhERERj\nUlQiIqIxKSoREdGYFJWIiGhMikpERDQmRSUiIhqTohIREY3p+aIiaaakeyUNSJrb7XwiInZmPV1U\nJI0DvgocBxwEnCzpoO5mFRGx8+rpogIcDgzYXmf7OWARMKvLOUVE7LR6vahMBB5o2R4ssYiI6IJd\nu53A70ltYt6ikTQHmFM2n5J0b0ez6q59gUe6nURd+ofZ3U7hxaSn/nYAzGv3T3Cn1VN/P31yq/92\nr63TqNeLyiAwuWV7EvDg8Ea25wPzt1dS3SRple3+bucRWy9/u96Wv1+l129/rQSmSjpA0u7AScCS\nLucUEbHT6ukrFdubJJ0OLAPGAQtsr+1yWhERO62eLioAtpcCS7udx4vITnGbbweVv11vy98PkL1F\nv3ZERMQ26fU+lYiIeBFJUdlBZLqa3iVpgaSHJd3Z7Vxi60iaLOl6SXdLWivpjG7n1G25/bUDKNPV\n/B/gHVTDrFcCJ9u+q6uJRS2S3go8BVxm+43dzifqkzQBmGD7Vkl7AquBE3bmf3u5UtkxZLqaHmb7\nRmBjt/OIrWd7ve1by/qTwN3s5LN6pKjsGDJdTUSXSZoCHAKs6G4m3ZWismOoNV1NRHSGpFcA3wI+\nZfuJbufTTSkqO4Za09VERPMk7UZVUL5p+9vdzqfbUlR2DJmuJqILJAm4BLjb9gXdzufFIEVlB2B7\nEzA0Xc3dwOJMV9M7JF0O/AT495IGJZ3a7ZyithnAB4GjJa0py/HdTqqbMqQ4IiIakyuViIhoTIpK\nREQ0JkUlIiIak6ISERGNSVGJiIjGpKhEFJL+d402n5L0sg7nMW2kYamSjpR0TcPf1/g5Y+eVohJR\n2H5LjWafAraqqJRZpLfGNGCnftYheleKSkQh6anyeaSkGyRdJekeSd9U5ZPAa4DrJV1f2h4j6SeS\nbpV0ZZkDCkn3Szpb0k3AiZIOlPR9Sasl/S9JbyjtTpR0p6TbJN1YZkQ4B/hAeZDuA6Pk+/LyLpaV\nkn4qaVaJr5B0cEu7GyQdNlL7iCb1/DvqIzrkEOBgqjnUfgzMsH2RpE8DR9l+RNK+wFnA220/Lelz\nwKepigLAM7b/A4Ck64CP2r5P0hHA14CjgbOBY23/StJ4289JOhvot336GDn+NfBD2/9Z0njgFkk/\noHr1wfuBeeV9H6+xvVrS347QPqIxKSoR7d1iexBA0hpgCnDTsDbTgYOAH1dTQLE71XQrQ64ox78C\neAtwZWkH8JLy+WPgUkmLga2djPAY4N2S/qJsvxTYH1gMLAfmURWXK8doH9GYFJWI9p5tWX+e9v9W\nBCy3ffII53i6fO4CPGZ72vAGtj9arlzeCayRtEWbUQj4M9v3brFD+jdJfwR8APjIaO0l7bcV3xkx\nqvSpRGydJ4E9y/rNwAxJfwAg6WWSXj/8gPJ+jZ9LOrG0k6Q3l/UDba+wfTbwCNUrDFq/YzTLgE+U\nmXKRdEjLvkXAXwKvsn1HjfYRjUhRidg684FrJV1vewPwIeBySbdTFZk3jHDcnwOnSroNWMsLr3v+\nkqQ7JN0J3AjcBlwPHDRWRz1wLrAbcHs5/tyWfVdRvQJhcc32EY3ILMUREdGYXKlERERjUlQiIqIx\nKSoREdGYFJWIiGhMikpERDQmRSUiIhqTohIREY1JUYmIiMb8P9OrXRxDhK9cAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10c415cf8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(y_train)\n",
    "plt.xlabel('interest level')\n",
    "plt.ylabel('Number of Occurences')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "各类样本分布并不均衡."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2    34284\n",
       "1    11229\n",
       "0     3839\n",
       "Name: interest_level, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看稀有事件的数目\n",
    "y_train.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.5854471271864323"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 计算 min_child_weight 最优值\n",
    "1 / np.sqrt(3839/49352)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "该结果可作为 min_child_weight 调参时的参考值."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Step I: 固定学习率 learning_rate 为 0.1, 初步确定弱学习器数目 n_estimators, 其他参数采用适当值."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 定义模型训练函数\n",
    "# 直接调用 xgboost 内嵌的交叉验证(cv), 可对连续的 n_estimators 参数进行快速交叉验证\n",
    "def model_fit(alg, X_train, y_train, cv_folds=None, early_stopping_rounds=10):\n",
    "    xgb_param = alg.get_xgb_params()\n",
    "    xgb_param['num_class'] = 3\n",
    "    \n",
    "    xgtrain = xgb.DMatrix(X_train, label=y_train)\n",
    "    # 直接调用 xgboost\n",
    "    cv_result = xgb.cv(params=xgb_param, dtrain=xgtrain, num_boost_round=alg.get_params()['n_estimators'], \n",
    "                       folds=cv_folds, metrics='mlogloss', early_stopping_rounds=early_stopping_rounds)\n",
    "    cv_result.to_csv('1_nestimators.csv', index_label='n_estimators')\n",
    "    \n",
    "    # 最佳参数 n_estimators\n",
    "    n_estimators = cv_result.shape[0]\n",
    "    \n",
    "    # 采用交叉验证得到的最佳参数 n_estimators, 训练模型\n",
    "    alg.set_params(n_estimators=n_estimators)\n",
    "    alg.fit(X_train, y_train, eval_metric='mlogloss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 本项目根据 Owen Zhang 调参说法, 行采样参数保持为 1 不变\n",
    "xgb1 = XGBClassifier(n_jobs=8, \n",
    "                     learning_rate=0.1, \n",
    "                     n_estimators=1000, \n",
    "                     max_depth=5, \n",
    "                     min_child_weight=1, \n",
    "                     gamma=0, \n",
    "                     subsample=1, \n",
    "                     colsample_bytree=0.4, \n",
    "                     colsample_bylevel=0.4, \n",
    "                     objective='multi:softprob', \n",
    "                     seed=3)\n",
    "# 5 折交叉验证时采用分层抽样\n",
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# 调用 xgboost 内嵌交叉验证, 初步调优 n_estimators\n",
    "model_fit(xgb1, X_train, y_train, cv_folds=kfold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "522"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb1.get_params()['n_estimators']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "初步确定弱学习器数目为 522."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 读取结果\n",
    "cv_result = pd.DataFrame.from_csv('1_nestimators.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEXCAYAAABCjVgAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8XGXZ//HPNZO9SbqlLU1TaCll\nKVvZQVB2KKCAomyi4kL19xNcUUAfEfi54OOGPioICLiCKKiIFfoARSqytaxtWbrQ0nRv2qbNvl2/\nP+6TZJpOmmmaySSZ7/vFec1Z7jnnOtMh19z3fc59zN0REREBiGU6ABERGTiUFEREpIOSgoiIdFBS\nEBGRDkoKIiLSQUlBREQ6KCmIyC4xs3eb2ZuZjkPSQ0lhEDKz5WZ2WpqPcYOZ/S6dx0gnM3vAzG7v\nsu6vZvazhOXxZnaHma02sxozW2Zm95jZ/tH2SWbm0bYaM1tnZr8ws9w0x+5mtk86j7Erusbj7nPd\nfb80HeseM/tWOvYtqVFSkKHqs8AFZnYygJldBBwGXBstjwb+AxQB7wZKgMOBfwGnd9nXCHcvBg4G\njov2LQOQmcUzHcNgp6QwxJjZFWa2xMw2mdlDZlaesO0MM3vTzKqjX7z/MrNP9eIYB5jZk2a2xcwW\nmtm5CdvONrNFZrbNzFaZ2dXR+jIzezh6zyYzm2tmO3z/zOw2M/tBl3V/M7MvRfPXRPvdFp3Lqcli\ndPe1wJeBO8xsT+CnwKfdvSYq8kVgK/ARd1/qwRZ3v9vd/6ebfa4H/heYluJnMdzMfmNmG8xshZn9\nV/s5m9k+0edfbWYbzeyP0fqnore/EtVOLkryGV1uZv82sx+Y2WYze9vMzkoWc5f3DTezX5nZmugz\n/Fb7H9FdicfMTjKzyoT9Ljezr5jZq2ZWGx1jnJn9M/p3eszMRiaU/5OZrY2O9ZSZHRitnwl8GPhq\ndKy/p/AZ32Nmt5rZLDOrBU7u7jsoKXJ3TYNsApYDpyVZfwqwkfCLNx/4H+CpaFsZ4Y/gB4Ac4PNA\nM/Cpbo5xA/C7JOtzgSXA14C86JjbgP2i7WuAd0fzI4HDo/nvArdF788l/Dq3JPt/D7CyfVu0j3qg\nHNgv2lYebZsETOnhs3o0+kx+3WX9s8ANPbx3EuBATrRcDrwCfCLFz+I3wN8ItZBJwFvAJ6Nt9wJf\nJ/wwKwBOSDiuA/vsJK7Lo3+7K4A48H+A1ck+zy7v+yvwS2AYMBZ4npAodyke4CSgssv38VlgHDAB\nWA+8SKiZ5QNPAN9MKP+J6DPJB24BXk7Ydg/wrV34vt0DVAPHJ8Se9DuoKbVJNYWh5cPAXe7+ors3\nAtcBx5nZJOBsYKG7P+juLYRfzmt7cYxjgWLgZndvcvcngIeBS6LtzcA0Myt1983u/mLC+vHAXu7e\n7KFdOtnAW3MJf4TeHS1/EHjG3VcDrYQ/JNPMLNfdl7v70h7inQuMBrr2j5SRcP5mdm70S3Sbmc3u\nUnajmW0BVgG1wJ97+iyiX+AXAde5+zZ3Xw78EPhIwuexFyHBNbj7v3s4j65WuPsd7t4K/Jrw2Y7r\nrrCZjQPOAr7g7rUeaj0/Bi7uo3j+x93Xufsqwmf+nLu/FH0P/0JIEAC4+13RZ9JI+PFxqJkN72a/\nPX3fAP7m7k+7e5u7N9D9d1BSoKQwtJQDK9oXPDSVVBF+vZUTfmW3b3OgsusOUjzGSndvS1i3IjoG\nwAWEBLQiao44Llr/fcIvvtkWOnSvTbbzKK776Pyf/lLg99G2JcAXCH9I1pvZfZbQPNaVmU0FrgZ+\nAfzQtu8griL8IW0/7kPuPoLQrJTXZVdl0bYi4GngkRQ+i7JoPyuSbAP4KmDA81GTyCe6O49udCQ0\nd6+LZot3Un4vwq/uNVHy20KoNYzto3jWJczXJ1kuhtDmb2Y3m9lSM9tKqGVA+LyS6en7Bgnf60h3\n30FJgZLC0LKa8D8/AGY2jPAreRWhSl2RsM0Sl3fxGBO79AfsGR0Dd3/B3c8j/LH5K3B/tH6bu3/Z\n3fcG3gd8qbv+AEJTxgfNbC/gGOCB9g3u/gd3PyE6Twe+l2wH0fndSWieuIrwC/+ahCKPA+dbkn6N\n7rh7PaG54jgzK+vhs9hI56/vrttw97XufoW7lwOfBn5h6b3iaCXQSJTgoqnU3Q/s53guBc4DTgOG\nE5rVICQkCP+miXb6fUv2nu6+g5IaJYXBK9fMChKmHOAPwMfNbLqZ5QPfIVTjlwP/AA42s/Ojsp8F\n9ujhGLEux8gHniP8gf2qmeWa2UmEP/L3mVmemX3YzIa7ezOhD6MVwMzeG3VmWsL61mQHdfeXgA2E\nP+qPuvuWaB/7mdkpURwNhF+gSfdBaGcvA74T/cr8ZBTz/tH2HxHam39rZlMsKAGmd/dhRMf9COFX\netXOPouoWed+4NtmVhIluC8RNWOZ2YfMrD0pbyb8YWs/l3XA3t3F0RvuvgaYTagxlZpZLDrvE/s5\nnhJCcqoi1Ly+02V712N1+xkn2/nOvoOSokx3amja9YlQ5fYu07eibZ8BlgKbCG2vFQnvm0Ho7Kwm\nNKk8Q7j6JtkxbkhyjMpo24GESzergUXA+6P1eYSmlc2E/xlfIOqwJDTLLCf8D14JfKOHc/xGdMwP\nJaw7hNA5ui3h/MqTvHcisAU4tsv6bxLau9s7scuBXxFqUTXR5/Zr4IBo+6Qohppo2hKd91EJ+0z6\nWUTbRhKSwAbCL/XrgVi07b8Jv3bbjzsz4X2fiWLaAlyY5PwuB/7dZd1OO6ejMsOBW6PPvxp4Cbh4\nV+MheUfzaQnLvyOhEx/4FPBYNF9M6HzfRmgG+mhi7MBU4OXoWH9N4TO+h+07prv9DmpKbWr/n0Oy\nTFQdrwQ+7O5zMh2PiAwMaj7KImZ2ppmNiJpBvkZox302w2GJyACipJBdjiM0DWwktMue76HzVIYI\n6xySo+v07p7fLYKaj0REpJNqCiIi0iEn0wHsqrKyMp80aVKmwxARGVTmz5+/0d3H9FRu0CWFSZMm\nMW/evEyHISIyqJjZip5LqflIREQSKCmIiEgHJQUREekw6PoURER6o7m5mcrKShoaGjIdSloVFBRQ\nUVFBbm7vnhqrpCAiWaGyspKSkhImTZpEGJdx6HF3qqqqqKysZPLkyb3ah5qPRCQrNDQ0MHr06CGb\nEADMjNGjR+9WbUhJQUSyxlBOCO129xyzJim8vXwpcx77B22tbT0XFhEBLvrlM1z0y2cyHUa/ypqk\nsH7uPZz870upr6/JdCgikoW2bNnCL37xi16995ZbbqGurq7ngn0ga5KC5RYB0FCnpCAi/W+wJIWs\nufrI8kJSaFRNQUQy4Nprr2Xp0qVMnz6d008/nbFjx3L//ffT2NjI+9//fm688UZqa2u58MILqays\npLW1lW984xusW7eO1atXc/LJJ1NWVsacOel9JlbWJIV4fkgKTfW1GY5ERDLtxr8vZNHqrT2WW7Qm\nlEmlX2FaeSnffN+B3W6/+eabWbBgAS+//DKzZ8/mz3/+M88//zzuzrnnnstTTz3Fhg0bKC8v5x//\n+AcA1dXVDB8+nB/96EfMmTOHsrKyFM+w97Km+SiePwyA5gbVFEQks2bPns3s2bM57LDDOPzww3nj\njTdYvHgxBx98MI899hjXXHMNc+fOZfjw4f0eW9bUFHKipKCagojs7Bd9ovYawh8/fVyfHt/due66\n6/j0pz+9w7b58+cza9YsrrvuOs444wyuv/76Pj12T7KmppBTEJJCS6OSgoj0v5KSErZt2wbAmWee\nyV133UVNTWi5WLVqFevXr2f16tUUFRVx2WWXcfXVV/Piiy/u8N50y5qaQl5hMQAtDf3Tgy8ikmj0\n6NEcf/zxHHTQQZx11llceumlHHdcqIEUFxfzu9/9jiVLlvCVr3yFWCxGbm4ut956KwAzZ87krLPO\nYvz48epo7it5haGm0NakmoKIZMYf/vCH7ZY///nPb7c8ZcoUzjzzzB3ed9VVV3HVVVelNbZ2WZMU\n8gtCTaFVSUFEUtTXfQmDQdb0KeQNC0nBG+szHImIyMCVNUmhMOpToFl9CiIi3cmapJCbV0CrG66k\nICLSrbQlBTO7y8zWm9mCbrabmf3UzJaY2atmdni6YokOSAP5WPPQfuqSiMjuSGdN4R5gxk62nwVM\njaaZwK1pjAWABsvHWlRTEJEU3X1OmLJI2pKCuz8FbNpJkfOA33jwLDDCzManKx6AJssn1qKagoj0\nv96Oknr22WezZcuWNESUXCb7FCYAKxOWK6N1OzCzmWY2z8zmbdiwodcHbGiN0bZlZc8FRUT6WHdJ\nobW1dafvmzVrFiNGjEhXWDvIZFJI9sw4T1bQ3W939yPd/cgxY8b0+oAteSUUFBX3+v0iIr2VOHT2\nUUcdxcknn8yll17KwQcfDMD555/PEUccwYEHHsjtt9/e8b5JkyaxceNGli9fzgEHHMAVV1zBgQce\nyBlnnEF9fd9fYp/Jm9cqgYkJyxXA6nQesCVWQE6bmo9Est4/r4W1r/Vcbu2r4TWVfoU9Doazbu52\nc+LQ2U8++STnnHMOCxYsYPLkyQDcddddjBo1ivr6eo466iguuOACRo8evd0+Fi9ezL333ssdd9zB\nhRdeyAMPPMBll13Wc2y7IJM1hYeAj0ZXIR0LVLv7mnQesCVeQK6SgogMAEcffXRHQgD46U9/yqGH\nHsqxxx7LypUrWbx48Q7vmTx5MtOnTwfgiCOOYPny5X0eV9pqCmZ2L3ASUGZmlcA3gVwAd78NmAWc\nDSwB6oCPpyuWdi2xAoa1rU/3YURkoNvJL/rttNcQPv6PPg9h2LBhHfNPPvkkjz32GM888wxFRUWc\ndNJJNDTs+AM2Pz+/Yz4ejw+u5iN3v6SH7Q58Nl3HT6Y1p5B8V01BRPrfzoa/rq6uZuTIkRQVFfHG\nG2/w7LPP9nN0nbJmQDwAzykgzxszHYaIZKHEobMLCwsZN25cx7YZM2Zw2223ccghh7Dffvtx7LHH\nZizOrEoKbTmFFCopiEiGdB06u11+fj7//Oc/k25r7zcoKytjwYLOASKuvvrqPo8PsiwpkFNIAY14\nWxsWy5phn0Skt9LQlzDQZddfxrwicqyNxibVFkREksmqpGC5hQA01NZkOBIRyYRwfcvQtrvnmFVJ\nIZZXBEB9vZKCSLYpKCigqqpqSCcGd6eqqoqCgoJe7yOr+hQsL1wX3KSkIJJ1KioqqKysZHfGTxsM\nCgoKqKio6PX7syopPPz6Fo4Amur0nGaRbJObm7vdHcSSXFY1H30w9z8ANNVvzXAkIiIDU1Ylhbz3\nfA6ApjolBRGRZLIqKRQUhzHJm1VTEBFJKruSwrDhALQqKYiIJJVVSaGoZCQAbQ3JB6USEcl22ZUU\nikNNgUYlBRGRZLIqKVhOHg3kQqPuUxARSSarkgJAHUXEmlVTEBFJJuuSQr0VEm/WzWsiIslkXVJo\niA8jt0XNRyIiyWRdUmiOF5HbWpfpMEREBqSsSwrrGvOINalPQUQkmaxLCqWxBkpQTUFEJJmsSwqt\no6ZSZHrymohIMlmXFDyvmCKvH9IP2hAR6a2sSwrkl1BoTdTVN2Q6EhGRASfrkoIVlABQu21LhiMR\nERl40poUzGyGmb1pZkvM7Nok2/cys8fN7FUze9LMev8MuRTFo6RQX1Od7kOJiAw6aUsKZhYHfg6c\nBUwDLjGzaV2K/QD4jbsfAtwEfDdd8bTLKQyD4jXUqKYgItJVOmsKRwNL3H2ZuzcB9wHndSkzDXg8\nmp+TZHufyy0sBaCxTjUFEZGu0pkUJgArE5Yro3WJXgEuiObfD5SY2eiuOzKzmWY2z8zmbdiwYbeC\nyouGz25WUhAR2UE6k4IlWdf1OtCrgRPN7CXgRGAV0LLDm9xvd/cj3f3IMWPG7FZQhcPCIzkffmHx\nbu1HRGQoyknjviuBiQnLFcDqxALuvhr4AICZFQMXuHtaf8IXlYakcCrPp/MwIiKDUjprCi8AU81s\nspnlARcDDyUWMLMyM2uP4TrgrjTGA0Dx8DIA2sYdku5DiYgMOmlLCu7eAlwJPAq8Dtzv7gvN7CYz\nOzcqdhLwppm9BYwDvp2ueNrFC0poJo416OojEZGu0tl8hLvPAmZ1WXd9wvyfgT+nM4YdmFFDMbFG\nJQURka6y7o5mgNp4CTlNuvpIRKSrrEwK9fES8pu3ZjoMEZEBJyuTQlNOKYWtSgoiIl1lZVJozh9O\nUZue0ywi0lVWJoXF1XFKvEbPVBAR6SIrk8KUnA2UUkdNQ1OmQxERGVCyMinYPqcSM6d6c1WmQxER\nGVCyMinkDBsJQF317g2uJyIy1GRlUsgrDgOx1m9VTUFEJFFWJoXC0pAUGrZuynAkIiIDS1YmhaIo\nKTTXqqYgIpIoK5NC8cjwTIbW2s0ZjkREZGDJyqRQUBJqCq8vW5HhSEREBpasTArkFtDguezvyzId\niYjIgJKdSQHYGh9BTmFJpsMQERlQsjYp1OaMpKBJVx+JiCTK2qTQkDeK4hZ1NIuIJMrapNBcWMaI\nti0aFE9EJEHWJgUvKmMU1VTXaVA8EZF2WZsU4iVjybNWNm3amOlQREQGjKxNCrml4wDYWrU6w5GI\niAwcWZsUCkfuAUDdpjUZjkREZODI2qRQMno8AI3V6zMciYjIwNFjUjCzKWaWH82fZGafM7MR6Q8t\nvUpHlwPQum1dhiMRERk4UqkpPAC0mtk+wK+AycAfUtm5mc0wszfNbImZXZtk+55mNsfMXjKzV83s\n7F2KfjfEi8vCTI0etCMi0i6VpNDm7i3A+4Fb3P2LwPie3mRmceDnwFnANOASM5vWpdh/Afe7+2HA\nxcAvdiX43RLPpdpKiNdr+GwRkXapJIVmM7sE+BjwcLQuN4X3HQ0scfdl7t4E3Aec16WMA6XR/HCg\nXy8FqomPJL9Rl6SKiLRLJSl8HDgO+La7v21mk4HfpfC+CcDKhOXKaF2iG4DLzKwSmAVclcJ++0xD\nfhnDmpQURETa9ZgU3H2Ru3/O3e81s5FAibvfnMK+LdnuuixfAtzj7hXA2cBvzWyHmMxsppnNM7N5\nGzb0XR9A47DxlHkVjS2tfbZPEZHBLJWrj540s1IzGwW8AtxtZj9KYd+VwMSE5Qp2bB76JHA/gLs/\nAxQAZV135O63u/uR7n7kmDFjUjh0ikrLGcdm1m6u7bt9iogMYqk0Hw13963AB4C73f0I4LQU3vcC\nMNXMJptZHqEj+aEuZd4BTgUwswMISaHfLgfKHzWRHGtjw9rK/jqkiMiAlkpSyDGz8cCFdHY09yi6\nYulK4FHgdcJVRgvN7CYzOzcq9mXgCjN7BbgXuNz7cdjSYWP2BOCuWXP765AiIgNaTgplbiL8YX/a\n3V8ws72Bxans3N1nETqQE9ddnzC/CDg+9XD71og9JgPwPn8S+HSmwhARGTB6TAru/ifgTwnLy4AL\n0hlUfykYFbo8Wkftk+FIREQGhlQ6mivM7C9mtt7M1pnZA2ZW0R/BpV3RKBrJI6dGI6WKiEBqfQp3\nEzqIywn3Gfw9Wjf4mbElZwwF9Rr/SEQEUksKY9z9bndviaZ7gD68LjSzKltHUNy4To/lFBEhtaSw\n0cwuM7N4NF0GDJkBg4bnNDPeqthQ05jpUEREMi6VpPAJwuWoa4E1wAcJQ18MCTn7ncl4NrFi3eZM\nhyIiknGpDHPxjruf6+5j3H2su59PuJFtSCgevy8xczaufCvToYiIZFxvn7z2pT6NIoNGTNwfgMef\nfjbDkYiIZF5vk0Kywe4GpZyycI/C0cO3ZDgSEZHM621SGDqX6hSOpDZWTMG2FZmOREQk47q9o9nM\ntpH8j78BhWmLqL+ZUV0wkZG1lbS1ObHYkKkEiYjssm6TgruX9GcgmdQ0fBITa15k1ZZ6Jo4qynQ4\nIiIZ09vmoyGlYOw+VNgGXl+lp7CJSHZTUgBG7nUgOdbGumULMh2KiEhGKSkA+eUHA7DoZV2WKiLZ\nTUkBoGxfWokzLa4nsIlIdktl6OxtZra1y7QyGk577/4IMu1y8tgSG874xmVsa2jOdDQiIhmTypPX\nfgSsBv5AuBz1YmAP4E3gLuCkdAXXn16NH8j+ra+zcPVWjt17dKbDERHJiFSaj2a4+y/dfZu7b3X3\n24Gz3f2PwMg0x9dvjj323VTYRl5dujLToYiIZEwqSaHNzC40s1g0XZiwbcjc2Vz4xl8BqFr6YoYj\nERHJnFSSwoeBjwDro+kjwGVmVghcmcbY+tfH/hZeK+fR1jZkcp2IyC7psU/B3ZcB7+tm87/7NpwM\nKh5LTeEEDq5ZwrKNNewzNmtu6BYR6ZDK1UcV0ZVG681snZk9YGYV/RFcf3N3Dost4VO/npfpUERE\nMiKV5qO7gYeAcmAC8Pdo3ZBTfOJVTLAqjhzVkOlQREQyIpWkMMbd73b3lmi6BxiT5rgywiYeDUDd\nsmdpaG7NcDQiIv0vlaSw0cwuM7N4NF0GVKWyczObYWZvmtkSM7s2yfYfm9nL0fSWmWX2STd7HEJL\nThHH2EKee3tTRkMREcmEVJLCJ4ALgbXAGuCDwMd7epOZxYGfA2cB04BLzGxaYhl3/6K7T3f36cD/\nAA/uWvh9LCcPm3QC74m9xlf//EpGQxERyYQek4K7v+Pu57r7GHcf6+7nAx9IYd9HA0vcfZm7NwH3\nAeftpPwlwL0pRZ1G8aolTI6tZVJsI+66NFVEsktvB8T7UgplJgCJtwdXRut2YGZ7AZOBJ7rZPtPM\n5pnZvA0bNuxqrLvm0vsA2Hvb8yxeX5PeY4mIDDC9TQqpPLMyWZnufnpfDPzZ3ZP27rr77e5+pLsf\nOWZMmvu4y/alNZ7Pe2KvMuu1Nek9lojIANPbpJBKu0olMDFhuYIwsF4yFzMAmo4AMCN+yIc4Pr6I\nO/+1ONPRiIj0q26TQjdDZm81s22EexZ68gIw1cwmm1ke4Q//Q0mOsx9hYL1nenkOfW/lC5RSywEt\nb/Dm2m2ZjkZEpN90mxTcvcTdS5NMJe6eyvAYLYSxkR4FXgfud/eFZnaTmZ2bUPQS4D4fSL26n3oM\nj+UyI2c+v39uRaajERHpN6k8T6HX3H0WMKvLuuu7LN+Qzhh6paAU2/tEzlgynxOfXc6Xz9iP4YW5\nmY5KRCTt9DjO7ux/DhNZy76s5L0/nZvpaERE+oWSQnf2fx/Ecrg4/2nWVDfoMZ0ikhWUFLpTPAb2\nncFlhc9gbc3cMfftTEckIpJ2Sgo7c/hHyanfyHlFr/KzJxZTVdOY6YhERNJKSWFnppwKI/bkitjf\naXPnZ3OWZDoiEZG0UlLYmXgOnPAl9mt5k/cNe4O7n17O2T95KtNRiYikjZJCT6ZfCqUT+PH42Zg5\nr6/dRk1jS6ajEhFJCyWFnuTkw/FfIKfyOS4cvRx3OPn7T2Y6KhGRtFBSSMXhH4XiPfhe2SOUDy9g\nQ00j9z3/TqajEhHpc0oKqcgtgOM/B8vn8tRFeQwvzOXaB1/jzB//K9ORiYj0KSWFVB3xcSidQM4j\n1/D0V95NUV6ct9bXMHvh2kxHJiLSZ5QUUpVXBGd9D9YvpPjFXzL3qydjwMzfzufE78/JdHQiIn1C\nSWFX7P9eOOB9MOfbjK5dysIbZxCPGSuq6jjqW49R35T0GUEiIoOGksKuMIP33gL5pfCXT1MYa+Wt\nb53V0fl84DcfYdHqrZmOUkSk15QUdtWwMnjfT2Dtq/CTQ4jHjP9cdyq//eTRuMPZP53LL/+1lObW\ntkxHKiKyy5QUeuOA98JxV8K2NfDcLwF499QxzP/G6YwsyuW7/3yDg294lLmLN2Q4UBGRXaOk0Fun\n3wSFo+CfX4UX7gRg1LA8Xrr+DO66/EgaW9r4yK+e54rfzFOTkogMGjaQnoKZiiOPPNLnzZuX6TCC\nlib44X5Qvwne+2M48hMdmxpbWvnVv9/mB4++SZtDaUEOt1w8nZP2HUssZhkMWkSykZnNd/cjeyyn\npLCbWhqjxLA59DUccfl2m6vrmvn98yv4/iNv4kBBboyvn30A7z+8guL8tD4NVUSkg5JCf0pMDGf/\nAI6+Yociza1tzHptDV978DVqo0tXzz20nHMPLeeEqWUU5Mb7O2oRySJKCv2tuQF+dEBoSnrX5+C0\nGyG2Y5eNu/PSyi08ML+S3z/XOX7SaQeM44xp4zj1gLGMLs7vz8hFJAsoKWRCaws8ck3oeC4aDV9c\nCLmF3RZvbGnluWWbePz1ddz7/EqaostY4wafPWUqx+49isP3HKlahIjsNiWFTHGHZ34Os78OFoMr\n58HoKSm8zVm0ZiuzF67jzrnLOpqYAEoKcvj48ZOVJESk15QUMu2tR+G+S0OSuOBOOOgDu/T2bQ3N\nzFu+ma/95TXWbW2gLeGfKW6wx/ACfvCh6Ry25wglCRHpkZLCQLDlHbj1XdC4DYaNgf/7HAwb3atd\nbW1oZt7yTTy7bBN3zl22XZIozs/hfYeWM218CdPKS9l/j1KG6comEUkwIJKCmc0AfgLEgTvd/eYk\nZS4EbgAceMXdL93ZPgdVUgBobYanvg//+m+I5cD7b4ODLgjjKO2G6vqQJJ5ZWsUf561kW8P2jwg1\ngzOmjWOfscXsM7aYKWOK2XtMsS6DFclSGU8KZhYH3gJOByqBF4BL3H1RQpmpwP3AKe6+2czGuvv6\nne130CWFdusWwq/OgKYaKBwJn54LIyb22e7dnTXVDSxavZVFa7Zy99NvU13fvF2Not2w/Dgn7TeW\nipGFVIwoZI/hhYwfXsC40gJGD8vTzXUiQ9BASArHATe4+5nR8nUA7v7dhDL/Dbzl7nemut9BmxQA\n2lrh+dvhkWvD8mk3hnsa8oal7ZBNLW2sqKpl6YZalm6oYUVVLY8sWMu2xha6+6c34LA9RzCutICy\n4nzKivMZXZwXzed1LBfn52C7WeMRkf4xEJLCB4EZ7v6paPkjwDHufmVCmb8SahPHE5qYbnD3R5Ls\nayYwE2DPPfc8YsWKFWmJud9sXgGzrobFswGDU68PySG/pF/DaGtzNtY0sqa6gbVbG1i3tYG11dG0\ntYFnl1UlrWkkyo0bk0YPY2RRHiOKcsPrsPA6siiXEUV5282PKMolN64ht0T620BICh8CzuySFI52\n96sSyjwMNAMXAhXAXOAgd99sASSsAAASMElEQVTS3X4HdU2hq5XPw+8vhIbNob/hxGvhmJlQMDzT\nkW2nqaWNzXVNbNjWyMaaRqpqmqiqbWRzXTNb6prYXNvM5romXli+qcckAhA3o3xkQZRIQsJoTyql\nBbkMy49TlJfT+ZqXQ1F+vOO1KDdOjhKLyC5JNSmks9exEkhsNK8AVicp86y7NwNvm9mbwFRC/8PQ\nN/FouHY5rJoP//o+zPlWmE68Bo75DBSNynSEAOTlxBhXGvocUuHu1DW1srmuiS11IWFsqu2cb39t\nTypzF2/otilrZ0YW5W6fPDqSSJyi/Og1yfaC3Di5cSM/J0ZuvHPKi8fIzbHtl+NGPGZqJpOskc6a\nQg6haehUYBXhD/2l7r4wocwMQufzx8ysDHgJmO7uVd3td0jVFLpa8wr85v1QXwUWhxO+AMd+tteX\nsQ42za1t1Da2UNvUSl1jC3VNrdQ2tVDXGL02tVKbbH1jK88s20h1fUvPB+ljRXlxpo4rIT9KKHnt\nCSUnmqL53ITXkIxsu3V5HcnJyIkb8ViMnFhISJ2vsfAaT74+vkN5JTTplPGagru3mNmVwKOE/oK7\n3H2hmd0EzHP3h6JtZ5jZIqAV+MrOEsKQN/5QuGYZrF0QLmOd+8MwHf5ROOoKGH9IpiNMq9x4LOp3\n6Lt9trU59c2dyaOmsYXGljaaWzunphaPXhPWtW6/rqmljab21/Z10XvDaysNzW1srW9h4erqlJrR\nBqqC3Bh7jioiZiGpxMyIxYyYhaa/mBkYxIywzQzrmKdj2aJlo3M74b/OMiSuj8q3z8eAjvd22Q90\nzFvCfNL9RAV2OG5sJ/tJUtaiwtvHEubpso/2PNxeLpF17L9ze+f72veR7Lxg33EllI/ofuicvqCb\n1way9W/Asz+Hl34H3haeDX3OD2HaeZCjQfMGI3enuWvC6ZgP61vbnJY2j14Tllvbl9l+fcf2Nlod\nWtvaEsqH7a3utCWUbfMw3/7a2gZtHm1LeI93lAvb29xpi8p6dD6d28LykvU1NDS3DurEOFBNGl3E\nk185uVfvzXhHc7pkVVJoV78ZXvo9PPH/oKUhdEof91k44uMwanKmoxMZFNwd98SEFuZJmHeiVwd8\nx+TneLQ+zHe8L/ozmjjf/r72+fbjtG/1Lvvx7uYT9jtxVCFjS1Lr2+tKSWEoamuDt5+EF34Fbzwc\n1k05NTQvTT0D8vqw3UVEhpSM9ylIGsRiMOWUMG1dDS/+JvQ5LH08bD/oAjjog7DPaZCTl9lYRWRQ\nUk1hsGttgRVPw4MzoWYdHRXVwz8aEsSkEyCmUVRFsp2aj7JRazMsexL++n+gdkNYF8+Doz4VEsSE\nw3d7ID4RGZyUFLJdUx0sfhQe/lJ4RChATkEYTmO/c8KNc6pBiGQNJQXpVL8FXv87LHwQlj4R1sVy\n4JCLYN8zYa8TsuYGOZFspaQgyTVUw5LH4I1ZsPAv4NFjP8fsD3u9C/Y6PryWlmc2ThHpU0oK0rPW\nZqicB+/8B1b8JySLdiMndSaIvd4FIyerP0JkEFNSkF3X2gLrXgsJ4qkfdPZFAGDhOdPttYmy/YjG\nIRCRQUBJQXZfWxtsfDNc8jrnu1C3sXNbLAf2ndGZJPY4WB3XIgOYkoL0PXfY/HaoSTx2Y3TZa/T9\nsThMOTkkiYqjoXx6vz80SES6pzuape+Zwai9w3TYZWFd9Sp455lQm3jlvu37JXKL4MAPhPsjJhwB\n4w6EeG5mYheRlKimIH2rtgpWvxgeHPTcbWEwv0T5JTD9wyFJTDgiJBh1YIuknZqPZGBwhy0rQpJY\n9SI8e2vnZbAQ+iYmv6czSZQfDiXjMhevyBClpCADV2sLbHgjShTz4aXfhudFtIvnw34zQoIonx4e\nPlQ4MnPxigwBSgoyuDTVwdpXOxPFggfpHIUesFi4A3vcQTBuGoydBsXj1PQkkiIlBRn86jbBmpdh\n9Usw5zvQ1vUZzBaeRnfoRSFJjJ0GYw+AgtKMhCsykCkpyNBUuxHWL4J1i2D9wvBEusQ+CgjNT3uf\n1FmjGDsNyvbVMyYkqykpSPZoa4PqlVGyWAjP3gZ1G5KXLSqDYz/TmSxG7KU7syUrKCmItDRB1ZKQ\nLNYvgqd/kqQJitBfUTgKjvw4jJoCo/eB0VOgaFT/xyySJkoKIt1p3AYb3gy1ivWvwwt3JE8WEO7U\nLhoNR88MiWL0PuHeivzi/o1ZZDcpKYjsqpYm2LwcNi2FqqWhlvHib3bss0g0bGxIGKMmd97tXTii\n30IWSZWSgkhfaqqFTctComhPGK/e333CsBhMOz8ki5GTwtDjoyZDSbn6MCQjlBRE+ktzPWx6OySN\nTUtDbWPBg9Cwpfv3FI6Egy/srF2M2htG7KkrpCRtBkRSMLMZwE+AOHCnu9/cZfvlwPeBVdGqn7n7\nnTvbp5KCDCqtLeHKqM3Lwwizm5eHBPLGw9vfxZ2oYAQcdMH2CWPkJMgt6MfAZajJ+CipZhYHfg6c\nDlQCL5jZQ+6+qEvRP7r7lemKQySj4jlRf8Nk4OTtt7lDXVVUw4imF+4Kl9PO+1X3+yweB0d9Ckon\nwPAJoYYxfKJGoJU+kc6hs48Glrj7MgAzuw84D+iaFESykxkMKwvTxKPDupO/1rm9blNCs9SycJVU\n7QaoWQdzvp18n/mlsP85IVGM2DPchzFiz5BA4hopX3qWzm/JBGBlwnIlcEyScheY2XuAt4AvuvvK\nrgXMbCYwE2DPPfdMQ6giA1DRqDBVHBGWT7qmc1tzA2xdBdWVUfPUCph/d0gkr9zbzQ4tPCVveEWo\nYZROCPPtNY6CERpLStKaFJJ9u7p2YPwduNfdG83sM8CvgVN2eJP77cDtEPoU+jpQkUEntyC6b2JK\n57pTvt4539IUksWWd6JpRUgci2fDin93v1+LhSFCuiaM0iiR5A1L1xnJAJHOpFAJTExYrgBWJxZw\n96qExTuA76UxHpHskZO3Y9JI1NYamqGqV8HWyuh1Fbz6R1j2ZPed4BASR/5w2P9sKNkjShwToxpI\nhQYkHOTSmRReAKaa2WTC1UUXA5cmFjCz8e6+Jlo8F3g9jfGISLtYHErLw8RRnetnfLdzvqUJtq3u\nTBjVldHrKlj8KLz8++73bzGYcmqoXQyv6KxplEaTrqQasNKWFNy9xcyuBB4lXJJ6l7svNLObgHnu\n/hDwOTM7F2gBNgGXpyseEdlFOXnRjXeTui/TUeOI+jaq2/s5KuHtJ8NNf8nEcsMotiXlUDo+4XV8\nSFQl46FguPo4MkA3r4lIejXXw9bV29c02pusVjwdtu/Q3RjJKYCKo6JkMX7HJFI8Tpfipijj9ymI\niACQW7jz/g0IV1NtWxOmrauj1zWh+WrrGlj0N2htTP7e4nHb1zCSJY/8UtU6UqSkICKZl1uQcJNf\nN9pv9utIGl1eN6+Ad56B+s1J9j9s++ap0vKEK6yiZFJUpnGpUFIQkcEi8Wa/8Yd0X66nWseKZ8J8\n1+HSYzlQvEe4oqpkj5AoEl9Ly8PrEL+fQ0lBRIaWVGodbW1Quz70c3QkjTWwbW14rVoKy/+dfFDD\nnILkSaOkfPvlQfrMDSUFEck+sVhnjWBnmuu3Txbbva6Fta/BW7OhOclVVnklnccoLU+eSIr3GHCX\n5yopiIh0J7ewc6Ta7riHp/klTRyrw+s7z4Q7y5MpHJWk1tE+H3WcDxvbb2NXKSmIiOwOs3AXd0Ep\njNm3+3LuoRN8W0JTVddmq/Wvh0Sy40GgeCycdiNMvyRtpwJKCiIi/cOsc5DDcQd2X66tFWo3JiSP\nhKQxYmL37+sjSgoiIgNJLA4l48LE9P4/fL8fUUREBiwlBRER6aCkICIiHZQURESkg5KCiIh0UFIQ\nEZEOSgoiItJBSUFERDoMuievmdkGYEUv314GbOzDcAaybDnXbDlPyJ5z1Xmmx17uPqanQoMuKewO\nM5uXyuPohoJsOddsOU/InnPVeWaWmo9ERKSDkoKIiHTItqRwe6YD6EfZcq7Zcp6QPeeq88ygrOpT\nEBGRncu2moKIiOyEkoKIiHTImqRgZjPM7E0zW2Jm12Y6nt1hZneZ2XozW5CwbpSZ/a+ZLY5eR0br\nzcx+Gp33q2Z2eOYi3zVmNtHM5pjZ62a20Mw+H60fiudaYGbPm9kr0bneGK2fbGbPRef6RzPLi9bn\nR8tLou2TMhn/rjKzuJm9ZGYPR8tD9TyXm9lrZvaymc2L1g3o729WJAUziwM/B84CpgGXmNm0zEa1\nW+4BZnRZdy3wuLtPBR6PliGc89Romgnc2k8x9oUW4MvufgBwLPDZ6N9tKJ5rI3CKux9KeNzWDDM7\nFvge8OPoXDcDn4zKfxLY7O77AD+Oyg0mnwdeT1gequcJcLK7T0+4J2Fgf3/dfchPwHHAownL1wHX\nZTqu3TynScCChOU3gfHR/HjgzWj+l8AlycoNtgn4G3D6UD9XoAh4ETiGcMdrTrS+43sMPAocF83n\nROUs07GneH4VhD+GpwAPAzYUzzOKeTlQ1mXdgP7+ZkVNAZgArExYrozWDSXj3H0NQPQ6Nlo/JM49\najY4DHiOIXquUZPKy8B64H+BpcAWd2+JiiSeT8e5RturgdH9G3Gv3QJ8FWiLlkczNM8TwIHZZjbf\nzGZG6wb09zenvw+YIZZkXbZcizvoz93MioEHgC+4+1azZKcUiiZZN2jO1d1bgelmNgL4C3BAsmLR\n66A8VzN7L7De3eeb2Untq5MUHdTnmeB4d19tZmOB/zWzN3ZSdkCca7bUFCqBiQnLFcDqDMWSLuvM\nbDxA9Lo+Wj+oz93McgkJ4ffu/mC0ekieazt33wI8SehHGWFm7T/eEs+n41yj7cOBTf0baa8cD5xr\nZsuB+whNSLcw9M4TAHdfHb2uJyT6oxng399sSQovAFOjKxzygIuBhzIcU197CPhYNP8xQvt7+/qP\nRlc2HAtUt1ddBzoLVYJfAa+7+48SNg3Fcx0T1RAws0LgNEJH7Bzgg1Gxrufa/hl8EHjCo4bogczd\nr3P3CnefRPj/8Al3/zBD7DwBzGyYmZW0zwNnAAsY6N/fTHfE9GOHz9nAW4R22q9nOp7dPJd7gTVA\nM+HXxScJ7ayPA4uj11FRWSNcebUUeA04MtPx78J5nkCoPr8KvBxNZw/Rcz0EeCk61wXA9dH6vYHn\ngSXAn4D8aH1BtLwk2r53ps+hF+d8EvDwUD3P6JxeiaaF7X93Bvr3V8NciIhIh2xpPhIRkRQoKYiI\nSAclBRER6aCkICIiHZQURESkg5KCiIh0UFIQSYGZTTezsxOWz7U+GoLdzL5gZkV9sS+R3aX7FERS\nYGaXE24mujIN+14e7XvjLrwn7mGsJJE+pZqCDClmNil6KM8d0cNqZkfDRiQrO8XMHolGsJxrZvtH\n6z9kZguiB948FQ2NchNwUfSwlIvM7HIz+1lU/h4zu9XCA4GWmdmJFh6E9LqZ3ZNwvFvNbJ5t/xCd\nzwHlwBwzmxOtuyR6MMsCM/tewvtrzOwmM3sOOM7MbjazRdEDWX6Qnk9Usk6mbwXXpKkvJ8JzJlqA\n6dHy/cBl3ZR9HJgazR9DGFcHwhADE6L5EdHr5cDPEt7bsUx46NF9hGEKzgO2AgcTfnTNT4ilfTiD\nOGHAu0Oi5eVEY+4TEsQ7wBjCKMZPAOdH2xy4sH1fhPH2LTFOTZp2d1JNQYait9395Wh+PiFRbCca\njvtdwJ+iZxj8kvDAE4CngXvM7ArCH/BU/N3dnZBQ1rn7a+7eRhjzpv34F5rZi4Qxjg4kPAWwq6OA\nJ919g4fnB/weeE+0rZUwYiyExNMA3GlmHwDqUoxTZKey5XkKkl0aE+ZbgWTNRzHCg12md93g7p8x\ns2OAc4CXzWyHMjs5ZluX47cBOWY2GbgaOMrdN0fNSgVJ9tPtwyKABo/6Edy9xcyOBk4ljDZ6JWEY\napHdopqCZCV33wq8bWYfgo6Hph8azU9x9+fc/XrC4x8nAtuAkt04ZClQC1Sb2TjC83jbJe77OeBE\nMyuLni1+CfCvrjuLajrD3X0W8AXCc51FdptqCpLNPgzcamb/BeQS+gVeAb5vZlMJv9ofj9a9A1wb\nNTV9d1cP5O6vmNlLhOakZYQmqna3A/80szXufrKZXUd4voABs9z9bzvukRLgb2ZWEJX74q7GJJKM\nLkkVEZEOaj4SEZEOaj6SIc/Mfk54NnCin7j73ZmIR2QgU/ORiIh0UPORiIh0UFIQEZEOSgoiItJB\nSUFERDr8fzia3wjnazGeAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a17376160>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 绘图, 查看模型在训练集和测试集上表现\n",
    "test_means = cv_result['test-mlogloss-mean']\n",
    "test_stds = cv_result['test-mlogloss-std']\n",
    "\n",
    "train_means = cv_result['train-mlogloss-mean']\n",
    "train_stds = cv_result['train-mlogloss-std']\n",
    "\n",
    "x_axis = range(0, cv_result.shape[0])\n",
    "\n",
    "plt.errorbar(x_axis, test_means, yerr=test_stds, label='test')\n",
    "plt.errorbar(x_axis, train_means, yerr=train_stds, label='train')\n",
    "plt.title('Log Loss vs XGBoost n_estimators')\n",
    "plt.xlabel('n_estimators')\n",
    "plt.ylabel('Log Loss')\n",
    "plt.legend(loc='best')\n",
    "plt.savefig('1_n_estimators.png')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAKvCAYAAAB3WjmUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3XuYHGWZ9/HvTcIEEpgJIBAixMhR\nEJTjrrKovCKgKCIuKyIeIp4WcJVl1RV1I7AKqKggoqxH8LS6y6KrsEgABREQ5AyCoJhwkGNEZiAh\nDEme94+qJpVOz0x3T89UH76f66or01VPVz/dM8pv7rnrqUgpIUmSJCmzVtkTkCRJktqJAVmSJEkq\nMCBLkiRJBQZkSZIkqcCALEmSJBUYkCVJkqQCA7IkSZJUYECWJEmSCgzIkiRJUoEBWepSETEvIlJE\n7N4Gc9k7n8shZc+lTBExP/8c9q9x7ND82Pur9k+LiKMj4vKI+EtEPJP/e1lEvC8i1q8an6q2JRFx\nR0R8MiJmTPR7HEtEvCUijil7HuMVER+LiDfU2F/5Wd+7hDlNj4jjy3htqdsYkCVp8pwEXA98IyIG\nKjsjYjPgK8AvgTML+zcGrgK+ANwJvBd4JfAu4Bbgs/nzqp0LvDTfDsofzwe+0/J31Li3AB0fkIGP\nAWsEZOAGss/9hsmdDgDTgU8Ce5fw2lJXmVr2BCSpV6SUlkfEO8hC8peAd+SHvgGsDbwzpZQKT/ke\nsBPwqpTSr6pO95OIOAF4TY2Xejil9JvC40si4nnA4RGxTkppWSvej9aUUhoCfjPmwA4SEesCy6p+\nNqWuZgVZ6nERMScivhcRj0TE0/mf4/8lItaqGrd5RJwbEU9ExOMR8f2I2CP/c/K8Fs1lw4j4SkT8\nOSKGI+JPEfHpiJhWNe4fIuKaiBiMiKX5uG8Vjq8VEZ+IiDsj4ql8vrdExAdHee2N89f89xrHXpC/\nzw/kj6dHxKkRsTAilkXEYxFxXUQcNtZ7TCn9jqya+/aIeH1EvAc4ADg2pXRP4TX3APYDvlYjHFfO\n9ZeU0vfGes3cIJCAFVXv7YiIuLnwPn4cEdvX+AxeHxFX55/3ExFxcUS8tGrMxhHxtYi4L/9ZejQi\nroyIV+XHLwNeCzyv2AYy2qQjYlFEnB8Rr46IG/Lv5+8j4og633fxXBERR0XETfl5/pr/TG9ZNW6X\n/DUr/5t4ICIuiIjN8+MJmAG8o/A+LsuPrdFiERFnR8ST+c/RRZG1vTwYER/Nj78kIn6d778r/yWq\n+nP9SkTcnp/nkYj4RUS8rDBmLvBo/vCThXmdXRizV0Rcmn//lkbEVRHx2qrXqrRm7RcR34qIR4Gl\nwLSxvr9SN7GCLPWwWPUn/D7g34BFwOuAU4GtgKPycTPI/vy/IfCvwB+BVwM/auFc1slfYyuyPxPf\nArwMOA7YmSxYkYeyH+Xb8cAy4HlkrQcVH8mPfQr4FVl19gXAzJFeP6X0aEScTxZ6PplSWlk4/E5g\nGPh+/vgLwNuATwA3koWlHYGN6ny7nyf78/zXyf4sfmFK6RtVY/bN//1pnecsioio/P/7esAryKrV\nP0wpPVMYdBxZ28d/kn3OG5F9bldHxB4ppT/k495C9t4XAIcB08g+48siYp+U0q/zU34X2BX4OHAX\n2ee9K6s+l6OAr5F9jw9u4P28mOwzOwV4GHg38M2I+ONIvzyM4D+AeWTV+38l+3meD1wVES9OKT2c\n/6xfDCwEjs5fbxbw/4BKv/dLgV+Q/bxWfqEaGuO11wbOA84CPkfWanJyRPQDfw98Brgf+Cfg7Ii4\nLaV0ff7cDfN/TwAeIvueHsyqz/8y4EGy/03+HPgm2V8lIA/NEfGK/H3dQtai8zTZ9+NnEXFYSqn6\nf8vfAi4g+zmfATzD2N9fqXuklNzc3LpwIwsCCdh9lDEn52P+pmr/V4CVwLb546Pyca+uGndWvn/e\nGHPZOx93yChj3peP+Yeq/R/J9++bP/6X/PHAKOf6GXBjE5/ZgcXXyvdNAf4MnFvYdyvw43F+f16a\nv9YyYHaN41/Nj29XtT/IihuVbUrV8TTC9n/AjMK4mWSVwQuqnr9FPqfv54/Xyt//LcBahXHrkYXH\nKwv7ngC+OMb7Ph9Y1MDntAh4CphT2LcO8BfgrAbO85L8czi2av/m+efwmfzxbvm4g8Y435PA2aP8\nrO9d2Hd2vu+NhX1TgUfy/bsU9m8ILAc+P8prT8mffwlwXmH/c/LzHV/jOVfn36/1qs5zK3AfEPm+\nefk5zqlxjjG/v25u3bLZYiH1tlcCt6eUrq3afzZZEKtUZV8BPJFS+nnVuP9s8VyWkF1QVj0XgH3y\nf3+b//tfEfGmiHhujXNdC7w4/7P0/nmVrh4XklXo3lnYtz8wm6yiVjz/ayLilPxP6uvWef6iY8h+\nCZkGvLyB5x1EVs2rbIM1xvwXsEe+vRz4ALA78PNY1a7yUmBdVn2+AKSU7iOrjlY+7+3I3v93U6Gq\nnlJ6Evgf4CURMT3ffS0wL7L2lpdExNoNvK/R3JRSurfw2svIKpjPa+AcryMLft+LiKmVjez7fTOr\nLmz7I/BX4DMR8Y8RsUMr3gCrfknJHqS0PH+tB1NKNxb2P0YWnFd7b/lcboiIZWQB+hmy79Ea7TDV\n8qr435L9kvdk4bVWkFWFNyf7Phf9T41TTdT3V2o7BmSpt21E9qfZag8Ujlf+fbjGuFr7xjOXh1JK\nq/WkppQeIQsEG+WPf0XWnjCVbFWG+yPitli9//dk4ENkVcMLgb/kvZejLnmXh5bvAgdHRKUdYx7Z\nZ3RRYegHyP4k/gayP7M/FhE/iYht6nmjEfEPwJuAY4HLgC9HxKZVwyqBsDoEXsaq8Hv+CC/xaErp\nuny7IqV0Rj7nvfL3A6u+tyN9/zeqc9xawAb540OBc8haIK4m+1y+ExGzRphnvf5SY9/TZAG/XpuS\n/dL3MKv/gvEM2c/JcwBSSoNkvxDeRNZ+8ru8B/mEcQbCpWnNiyOHgcdqjB0mq5IDEBHHkv1F4Rqy\ndoyXkH3/f059n8EGZO+9nv+tV9QaO1HfX6ntGJCl3vYXYLMa+2fn/y4ujKsOcJD1ZrZyLptGRBR3\nRsQmZGG4MhdSSv+bUtoHGCCr/N0P/CDvTyaltDyl9IWU0q5kf7I+jKx14KJCtXMk3yYLJ2+OiA2A\n1wPfyattlddfklL6ZErpBWSfwZFkoeVnY73JPAh/hSzofgk4In+9r1YNvTj/9/XFnSmlxyvhl9rB\ncSS35P++OP+38tyRvv+L6xy3kqziSkppcUrpmJTSXLJgfxzwRqqq1CVZTFbF3YtVv2AUt2eXbEsp\n3ZpSejNZaNyZrN99Pll7TxneClyWUjoypXRBSuma/Pu//lhPzP2V7PtUz//WK9a4eLLNv79SSxmQ\npd52KbBDROxatf/tZP+B/GX++HJg/YioXlLszS2ey3qsubbs2wvHV5NSejqldDnZBVcAu9QY83hK\n6Vyy9YU3BOaONomU0h1klbp3kl1INY0sNI80/uGU0tlk7Sbb1RHAzyILxEekzMJ8/gdHxLOfZx6A\nFgDvKa5WMA475/8+kv97NVlv71uLg/KVGl7Jqs/7TrIe5LcUf3nJ/2z/98DVKaWl1S+WUro3pfRl\nsqBf/PlqtPLbKueTVVGfW6iuF7dbq5+Qf39uTin9M/A45b2PlL/esyLiRWRtMkWVMavNK6W0hOxn\n+o3FdqDIVqp5K9kvmHc1NKGRv79SV3AVC6n7vTJfAqra/wFfJAugF0TEfOAestUijgK+mlKq/Efz\nHOCfyfo3P0HWO/kasv5cyKpT9XhJVYG44nKydomjgXPy+d5KVu37GPB/KaVLACLiRLKeyUvJ/sM+\nE/gg2Z/KL8/H/Ay4DbiO7Cr+55H1/N4D/KGOeX6LbMWD2cBVKaU7iwcj4hqywHULWXVue7Kr/WuG\nxcLz3kb2C8A/5sG44ivAIWStFr9MKVVaV95K1tpxSb5c10VkAbcfeBHwKmqvnrBpRLwk/3odsnD8\nCbKQ923IfnGIbEm7kyLiO2QBfyOyFUSWka2YQEppZUR8hGwVi/Mj4j/Ifmn4MNlnX1mqbIDsF6of\nAL8nu6BrD7KVFc4rzO1WsqB2JNl60CvzXwYmVErpyoj4GvDtvNXmV2Q975uR/ZzdmlL6akS8juzn\n/yfAn8hC9Rvz93px4ZS3AntHxIFk7QhPVP+ctND5wL9Ftu715WT9wvPJVtp49r/jKaUnIuIe4KCI\nuJSsfWNxSmkRWbX3YuCXEXEqWRvHUWSrrxxW3dpUrYHvr9Qdyr5K0M3NbWI2Vl2NPtI2Nx83hyz8\nLCb7j+bvyfp316o63xZkF+48QRbKziULyQl4/Rhz2XuMueydj9uQrNXgAbLAu4isD3Ra4VyvJQv3\n95NVzB4mW45qr8KYY4ErycLx02TB+BvA8+r87PrJVjZIwLtrHD+Z7GLBx8jC5N1kS79tNMo5Z5OF\n6YtGOP58spURzqvaPw14P3BF/vxnyNoefkW2wseGVeOrP9vhfH7fAraq8brvIrtI7WmyAP0TYIca\n4w4iuwHGU/k8LwH2rJrnV/NzDeaf3+/Jlo2bXhi3AfDfrPqzfxrje7EIOL/G/svI2g4a/d/FO/P3\n8WQ+xz+S/QK4W358O7IQ+Mf8+ONk1dd3VJ3nxcCvyUJ2qsyFkVexeHKE93DbWO+ZbBnGz5H9zD9F\n9ovFQfl5F1U9dx+yu/gty+dxduHYXmS/WFbe+9XA60b4/43dq/bX9f11c+uWrbKsiyQ1LCI+RrbW\n8JyU0v1lz0eSpFawxUJSXSLi/fmXvye76cEryVZG+J7hWJLUTQzIkuq1lKwPeS7Zn1vvJVvq7FMl\nzkmicNfAkaxMq98ZUZJGZYuFJKmjRcRY/yE7J6U0bzLmIqk7WEGWJHW6PcY4Xr3GrySNygqyJEmS\nVOCNQiRJkqQCWyyalN9RajbZmrCSJElqT+sDD6QG2iYMyM2bTbZouyRJktrb5sCf6x1sQG7eEwD3\n3Xcf/f39Zc9FkiRJVYaGhthiiy2gwb/4G5DHqb+/34AsSZLURbxIT5IkSSowIEuSJEkFBmRJkiSp\nwB5kSZKkkqSUWL58OStWrCh7Kh1pypQpTJ06lWz13dYxIEuSJJVgeHiYBx98kKVLl5Y9lY42ffp0\nNttsM/r6+lp2TgOyJEnSJFu5ciULFy5kypQpzJ49m76+vpZXQbtdSonh4WEeffRRFi5cyDbbbMNa\na7Wme9iALEmSNMmGh4dZuXIlW2yxBdOnTy97Oh1r3XXXZe211+aee+5heHiYddZZpyXn9SI9SZKk\nkrSq4tnLJuIz9LsiSZLUwZYOL2fuRy9g7kcvYOnw8rKn0xUMyJIkSVKBAVmSJEkqMCBLkiSpbnvv\nvTfHHHNMy843b9483vCGN7TsfK1gQJYkSZIKDMiSJEltIKXE0uHlTW0VzT4/pVTXHOfNm8fll1/O\n6aefTkQQESxatIjbb7+dAw44gPXWW49NN92Ut73tbSxevPjZ55177rnstNNOrLvuumy00Ua86lWv\nYsmSJRx//PGcc845/O///u+z57vsssta/dE2LOr9QLS6iOgHBgcHB+nv7y97OpIkqYMsW7aMhQsX\n8vznP//ZtXuXDi9nh/kXlTKf20/cn+l9Y98eY3BwkNe85jXsuOOOnHjiiQCsWLGCnXfemfe85z28\n/e1v56mnnuJf//VfWb58Ob/4xS948MEHmTNnDp/97Gc5+OCDeeKJJ7jiiit4+9vfDsC73vUuhoaG\n+Pa3vw3Ahhtu2NBd8Wp9lhVDQ0MMDAwADKSUhuo9pzcKkSRJUl0GBgbo6+tj+vTpzJo1C4D58+ez\n6667ctJJJz077lvf+hZbbLEFd911F08++STLly/njW98I8973vMA2GmnnZ4du+666/L0008/e752\nYECWJElqA+uuPYXbT9y/4ectHV7O7p+6FIDrPrFPXZXgWq/drOuvv55f/vKXrLfeemscu/vuu9lv\nv/3YZ5992Gmnndh///3Zb7/9OOSQQ9hggw2afs2JZkCWJElqAxHRVLgtmt43ddznaNTKlSs58MAD\n+cxnPrPGsc0224wpU6Zw8cUXc9VVV7FgwQLOOOMMPv7xj3PNNdfw/Oc/f1LnWi8v0pMkSVLd+vr6\nWLFixbOPd911V373u98xd+5ctt5669W2GTNmAFn4/7u/+ztOOOEEbrzxRvr6+vjxj39c83ztwIAs\nSZKkus2dO5drrrmGRYsWsXjxYo4++mgee+wxDjvsMK699lr+9Kc/sWDBAo444ghWrFjBNddcw0kn\nncR1113Hvffey3nnncejjz7K9ttv/+z5brnlFu68804WL17MM888U/I7NCBLkiSpAR/60IeYMmUK\nO+ywAxtvvDHDw8NceeWVrFixgv33358dd9yRD37wgwwMDLDWWmvR39/Pr371Kw444AC23XZbPvGJ\nT/D5z3+e17zmNQC85z3vYbvttmP33Xdn44035sorryz5HdqDLEmSpAZsu+22XH311WvsP++882qO\n33777fn5z38+4vk23nhjFixY0LL5tYIBWZIkqYNN75vKolNeW/Y0uootFpIkSVKBAbkDLB1eztyP\nXsDcj16w2u0kJUmS1HoGZEmSJKnAgNwBHh5a9uzXQ0+Vv/SJJElSNzMgd4BN1p/27NcvOfkXtllI\nkiRNIANyB4iIsqcgSZLUMwzIHWiH+RdZRZYkSZnhJXD8QLYNLyl7Nl3BgNwBpvdN5fYT919t38qV\nqaTZSJIkdTcDcoc6/9YHy56CJEnSuMydO5fTTjut7GmswYDcoT76P7e6LrIkSZp0e++9N8ccc0xL\nzvXb3/6W9773vS05VysZkDtErTYLSZKkdpNSYvny+gp4G2+8MdOnT5/gGTXOgNzhvGBPkqQukVJ2\nkV3D29JV5xhe2tw5Un3XNs2bN4/LL7+c008/nYggIjj77LOJCC666CJ23313pk2bxhVXXMHdd9/N\nQQcdxKabbsp6663HHnvswSWXXLLa+apbLCKCb3zjGxx88MFMnz6dbbbZhp/+9Kct+XgbMXXSX1FN\nm943lUWnvJZHn1jGHp++tOzpSJKkVnpmKZw0e3znOHXr5p73sQegb8aYw04//XTuuusudtxxR048\n8UQAfve73wHwkY98hFNPPZUtt9ySmTNncv/993PAAQfwqU99inXWWYdzzjmHAw88kDvvvJM5c+aM\n+BonnHACn/3sZ/nc5z7HGWecweGHH84999zDhhtu2Nx7a4IV5A40Y9rqv9dYRZYkSZNhYGCAvr4+\npk+fzqxZs5g1axZTpkwB4MQTT2Tfffdlq622YqONNuLFL34x73vf+9hpp53YZptt+NSnPsWWW245\nZkV43rx5HHbYYWy99dacdNJJLFmyhGuvvXYy3t6zrCBLkiS1g7WnZ5XcRg0vXVU5/tAfoa+Jnt61\nx98HvPvuu6/2eMmSJZxwwgmcf/75PPDAAyxfvpynnnqKe++9d9TzvOhFL3r26xkzZrD++uvzyCOP\njHt+jTAgd6DKBXs7zL/o2X07zL+I20/cn+l9fkslSepIEXW1OYyqb/r4z9GkGTNWf90Pf/jDXHTR\nRZx66qlsvfXWrLvuuhxyyCEMDw+Pep611157tccRwcqVK1s+39GYpiRJklS3vr4+VqxYMea4K664\ngnnz5nHwwQcD8OSTT7Jo0aIJnl1r2IPcoWot+2YvsiRJmmhz587lmmuuYdGiRSxevHjE6u7WW2/N\neeedx0033cTNN9/MW97ylkmvBDfLgCxJkqS6fehDH2LKlCnssMMObLzxxiP2FH/xi19kgw02YM89\n9+TAAw9k//33Z9ddd53k2TbHFosu8/jSYfuQJUnShNl22225+uqrV9s3b968NcbNnTuXX/ziF6vt\nO/roo1d7XN1ykWqsx/z44483N9FxsILcwWq1WXz3N6NfGSpJkrpM3ww4fjDbSrpAr9sYkLvMVy+7\nm7kfvcBeZEmSpCYZkDtcrSqyJEmSmmdA7lKuaCFJktQcA3IXmN43lUWnvJafHL1n2VORJEkNqHVR\nmhozEZ9h6QE5Io6KiIURsSwiro+Il40xfmZEnBkRD+bPuSMiDigcPy4ifhsRT0TEIxHxk4jYruoc\nl0VEqtp+OFHvcbJsu+n6qz22iixJUnuq3C1u6dKlJc+k81U+w+o78I1HqeuBRcShwGnAUcCVwPuA\nCyNih5TSGssxREQfcDHwCHAIcD+wBfBEYdgrgDOB35K9v08DC/JzLimM+zowv/D4qVa9L0mSpNFM\nmTKFmTNn8sgjjwAwffp0IqLkWXWWlBJLly7lkUceYebMmUyZMqVl544yS/sRcQ1wQ0rpyMK+O4Cf\npJSOqzH+H4EPAy9IKT1T52tsTBaoX5FS+lW+7zLgppTSMeOYez8wODg4SH9/f7Onabmlw8vZYf5F\nq+27/cT9XRtZkqQ2k1LioYceKmWd324yc+ZMZs2aVfMXjKGhIQYGBgAGUkpD9Z6ztNSUV4N3A06p\nOrQAGKmZ9vXA1cCZEXEQ8CjwA+AzKaWRbgo+kP/7WNX+wyPircDDwIXACSmlJxhBREwDphV2rT/S\nWEmSpLFEBJttthmbbLIJzzxTV91PVdZee+2WVo4ryiwrPgeYQhZQix4GZo3wnC2BVwLfBw4AtiFr\np5gKnFg9OLJfJb4A/DqldFvh0PeBhcBDwI7AycCLgX1Hme9xwCdHfUdtoLLsW7GKvMP8i6wiS5LU\npqZMmTIhIU/Na4fEVN3jETX2VaxF1i7x3rxifH1EzCZru1gjIANfBl4E7LXaC6b09cLD2yLiD8B1\nEbFrSumGEV77ZLKwXbE+WQ90R6gEZoOyJEnS6MpMSouBFaxZLd6ENavKFQ8Cz1S1U9wBzIqIvpTS\ncGVnRJxB1pLx8pTSWEH2BuAZsop0zYCcUnoaeLpw/jFOKUmSpE5U2jJveZi9njXbGvYFrhrhaVcC\nW0dEcd7bAg9WwnFkvgy8EXhlSmlhHdN5IbA2WQDveKPdXc+l3yRJkkZX9jrIXwDeHRFHRMT2EfFF\nYA5wFkBEfCciTi6M/yqwEXB6RGwbEa8FPkbWh1xxJvBW4C3AExExK9/Wzc+5VUTMj4jdI2Juvoby\nfwM3kgVwSZIk9bBSA3JK6UfAMWTrEd8EvBw4IKV0Tz5kDrBZYfx9wH7AHsAtwJeA01l9JYwjyVau\nuIysIlzZDs2PDwP7ABcBd+bnWAC8apSVMDqOVWRJkqTmlLoOcidr13WQi2qtiVzkBXuSJKmbNbsO\nctktFppA0/umsuiU145YSZYkSdKaDMg9zFYLSZKkNRmQe8BY/chzP3qBQVmSJClnQO4Ro4VkSZIk\nrWJAFmC7hSRJUoUBuYeMVUU2JEuSJBmQVcWQLEmSep0BucfYiyxJkjQ6A7IkSZJUYEDuQWPdQMQ2\nC0mS1MsMyKrJ9ZElSVKvMiD3MPuRJUmS1mRAliRJkgoMyD3OtZElSZJWZ0CWIVmSJKnAgKy6GJIl\nSVKvMCAL8II9SZKkCgOy6mYVWZIk9YJIKZU9h44UEf3A4ODgIP39/WVPp6WWDi9nh/kXjTrm9hP3\nZ3rf1EmakSRJUuOGhoYYGBgAGEgpDdX7PCvIWkM97RZWkyVJUrcyIKtphmRJktSNDMiqqd6L9gzJ\nkiSp2xiQNSJXtpAkSb3IgKxxs4osSZK6iQFZo2qk1WLuRy8wKEuSpI5nQFZLWU2WJEmdzoCsMU3v\nm8qiU15bdz+yIVmSJHUyA7Lq1shFe4ZkSZLUqQzInWB4CRw/kG3DS0qdiitbSJKkbmdA1oSxiixJ\nkjqRAbkTLBtc9fXKFeXNQ5IkqQcYkDvB2tNXff3QreXNI9doL7LLv0mSpE5iQO4EU9Ze9fXZB5Te\nhwyN9yLbbiFJkjqFAVlNa3T5N0mSpE5gQO5ET/217Bk0xSqyJEnqBAbkTtA3Az72wKrHd/+yvLnU\n4PrIkiSpmxiQO9FP398WfchFro8sSZK6hQG5Uz31eNkzWEO9IbmysoWrW0iSpHZkQO5Ud/287BnU\n5OoWkiSp0xmQO0V1H/IFx7bFradrcXULSZLUyQzImjCNtFxYRZYkSe3CgKwJ1WhfskFZkiSVzYCs\ntmJQliRJZYuUUtlz6EgR0Q8MDg4O0t/fP7kv/uidcObfrHr8sQeyHuU2tnR4OTvMv6jh591+4v5M\n75s6ATOSJEndbmhoiIGBAYCBlNJQvc+zgtyJBjYvewYNc51kSZLUKQzImjTNrG7hBXySJGmyGZA7\nUd8M+MBNqx4/fm95c2mC1WRJktTO7EFuUqk9yJCtf3zS7FWPO6APuVozfcn2JEuSpHrZg6yO00wl\n2ZYLSZI00QzI3WLxH8qeQVMMyZIkqd0YkDtV9a2nv/aKtrztdD28NbUkSWonBuRu0uH95I1Uk72h\niCRJmigG5G7yyO1lz2DcXOFCkiSVzYDcyarbLL65b8e2WTTLfmRJktRqBuRu0+FtFtB4FbnSbmHL\nhSRJagUDcrd58OayZ9ASlQv3vPOeJEmabAbkTlfdZnHnBeXNpU14AZ8kSRoP76TXpNLvpFdUfVc9\n6Mg7642lmTvvgXffkySpV3knPXW9Zle4sO1CkiQ1woCsjjKekGzbhSRJqoctFk1qqxaLip8fB7/5\nSvZ1F7ZY1NJs2wXYeiFJUrezxUKwzX6rvj5pdk+siTyeG4tYVZYkSbUYkLvJ5nuUPYNSjPfuewZl\nSZJUZEDuJhFlz6A0rbhFtUFZkiSBPchNa8se5B5Z7q0e4+lNrrBHWZKkzmYPsta8aUgPs6IsSZKa\nZUDudj38F4LK7aoNypIkqREG5G734E1lz6B0ragmg0FZkqReYQ9yk9qyB7miuhe5R/uQR2J/siRJ\nvaGje5Aj4qiIWBgRyyLi+oh42RjjZ0bEmRHxYP6cOyLigEbOGRHTIuKMiFgcEUsi4qcRsflEvL/S\nLX+67Bm0lVa0XlSqyVaUJUnqPqUH5Ig4FDgN+DSwC3AFcGFEzBlhfB9wMTAXOATYDngP8OcGz3ka\ncDDwZmAvYD3g/IiY0sK31x7+eEnZM2hLtl5IkqRaSm+xiIhrgBtSSkcW9t0B/CSldFyN8f8IfBh4\nQUrpmWbOGREDwKPA21JKP8pvYjn7AAAgAElEQVSPzwbuAw5IKY359/e2brEA2ywa1Iq2C7D1QpKk\ndtJsi0Wp/yXPq8G7AadUHVoA7DnC014PXA2cGREHkQXdHwCfSSmtqPOcuwFr5/sASCk9EBG35WPW\nSEoRMQ2YVti1/phvsJ0s+YsBeRSVtgsYX1guPs+wLElSZyr7v97PAaYAD1ftfxiYNcJztgReCXwf\nOADYBjiT7L2cWOc5ZwHDKaW/NvC6xwGfHOW9tLfbzoWXHVv2LDrCRIRlMDBLktQpSu9BzlX3eUSN\nfRVrAY8A700pXZ9S+iFZr/GRVeMaOWc9Y04GBgpbe1/QV33TkEtPgOMHstYL1a1VfcrghX2SJHWK\nsgPyYmAFa1ZtN2HNCnDFg8BdKaUVhX13ALPy9op6zvkQ0BcRG9T7uimlp1NKQ5UNeGLkt6Vu0qob\njhTtMP8iQ7IkSW2q1ICcUhoGrgf2rTq0L3DVCE+7Etg6Iopz3xZ4MKU0XOc5rweeKY6JiM2AHUd5\n3e5w0myryE2qBOVWhWVXv5AkqT21wyoWhwLfBf6R7OK795It2/bClNI9EfEd4M+VFS0iYgvgduBs\n4AyyHuRvAV9KKX26nnPmY74KvA6YBzwGnApsBOxWVZ0ead7tvYpF0V/uhjN2XfXYFS1aplWrXxTZ\nqyxJUmt05CoWACmlH0XERsB8YDPgNrKl1u7Jh8wBVhbG3xcR+wFfBG4hW//4dOAzDZwT4J+B5cB/\nAesClwLz6gnHHWf9ka471HgVL+iD1gTmyvMNypIklaP0CnKn6qgKMsA9V8O3X73qsVXkCdfq6rKB\nWZKkxnRsBVmTZLMXlT2DntOq5eIqXDZOkqTJUfYqFirLM0vLnkFPaeVycRVe5CdJ0sSwxaJJHddi\nAbBsCE7ZYtVj2yxKMREX9oEVZUmSqtliobGtNWX1x/5yVIpWt15UeJtrSZJawwpykzqygjy8JFsH\nucgqctuYqMoyGJglSb2p2QqyAblJHRmQYc2QbEBuS4ZlSZLGr9mA7EV6ve7Ru8qegWqYiNtbV3hx\nnyRJo7OC3KSuqSDvfDi84SvlzUd1mciKcoWVZUlSt7HFYpJ1bEAGe5G7wEQHZsOyJKkbuIqF1EMm\n4hbXRbXOZWiWJPUKK8hN6ugKcsUN34Wfvn/VY6vIHW8yWjGKDM2SpHbmRXpq3PavK3sGarGJvLiv\nFi/4kyR1I0s/vWxK3+qP/WtC16huwYCJrS57kxJJUjexxaJJXdFiAfD4fXDajqse22bR9Sa7DQMM\nzZKkcniRnpozfcOyZ6BJNtEX+NUy0vkNzpKkduR/mbS6vy6CTV9Y9iw0iSa7HaPI1gxJUjuyxaJJ\nXdNiAd5+WnVzlQxJUiexxUKts2zIgKyaitXmMlszKgzQkqSJYAW5SV1VQQZ4+kk4+bnZ1/t8El52\nbLnzUUcp48K/ehmiJal3eavpSdZ1Abm6zeKj98I6A+XNRx2rncNyhaFZknqDAXmSdV1ABliyGD63\nVfb1G78OL3pTufNRx+uEsFzN8CxJ3cOAPMm6MiBXV5HBC/bUEp0YlMdikJak9udFepLaVhlrL0+0\nRuZvmJakzmIFuUldWUGuOPddcNu52ddWkDWJuiE4N8sQLUmtZ4vFJOvqgLzo13B24cYRhmS1gV4O\nz2CAlqRm2GKh1tls57JnIK2hzDv+tYN636dBWpLGzwpyk7q6guzFeupwvRSc62VwltSLbLGYZF0d\nkAGGHoQvvGDVYwOyuoDBeWwGaUndxBYLtdY6VaF/+dMGZHW8Wm0ao+nFQG0rhyRZQW5a11eQbbOQ\n6tKLIbpRhmlJZbHFYpJ1fUCGNUPycX+GaeuVNx+pQxmix8eALalZtlho4t13DWy9T9mzkDrOWK0d\nBujRteKzMWRLaoQV5Cb1RAUZ1qwi22YhTRqDc/sxaEudxQqyJsfj98Im25c9C6kn1HtRoUF68nTS\n52yYl5pnBblJPVNBBi/Yk7qAIVqTod5QXvx5NMhrInmR3iQzIBuQpW5kkJZWd90n9mH3T10KGOY7\nkQF5kvV8QAZDstSjDNFS5yoGfmhN6G/nvwgYkCdZTwXkikfvhDP/ZtVjA7KkJhmypd42WUHai/Q0\n8QY2X/1xWlnOPCR1PC9AlNTOrCA3qScryABDD8IXXrDqsVVkSW3MgC21JyvI6i7r9NAvA5I6Xr2V\n6loM11LvsoLcpJ6tIIO9yJJUAgO7uokVZHWf6l5kSdKEG60abniWWsuArPE7abZVZEkqUaOtJAZq\naXQGZDWub0YWiItrI9uqI0kdYzy92WUx1GsyGZDVGot+Ddu9uuxZSJK6VJmh3nDee7xIr0k9fZFe\nhbegliRJNP5LhBfpSZIkqat1YtvOaNYqewLqMiuXlz0DSZKkcTEgq3l9M+D4QTjmtlX7TpmTtV5I\nkiR1KAOyxm/mFvDqU1Y9XvKX8uYiSZI0TgZktcbOh6/6+vSdrCJLkqSOZUBWa6zTD2/78arHJ802\nJEuSpI5kQFbrbPG3qz9euaKceUiSJI2DAVmt0zcDPnDzqsenbGEVWZIkdRwDslprvY1Xf+yNaCRJ\nUocxIGtiLby87BlIkiQ1xICs1uqbkd1uuuKHb4HjB2y1kCRJHcOALEmSJBUYkDU5XPZNkiR1CAOy\nWq9yC+r3XVH2TCRJkhpmQNbE2Wir1R9bRZYkSR3AgKyJ0zcDjrmt7FlIkiQ1xICsiTV9w9UfL3+6\nnHlIkiTVyYCsidU3Az6ycNXjzz7fNgtJktTWDMiaeFOnrf7Yu+tJkqQ2ZkDW5Dv5uVaRJUlS2zIg\na+JV310PshUtvMOeJElqQwZkSZIkqaD0gBwRR0XEwohYFhHXR8TLRhk7LyJSjW2dwphFI4w5szDm\nshrHfzjR77Wn1aoig2sjS5KktjO1zBePiEOB04CjgCuB9wEXRsQOKaV7R3jaELBdcUdKaVnh4R7A\nlMLjHYGLgf+uOs/XgfmFx081/AYkSZLUdcquIB8LfDOl9I2U0h0ppWOA+4AjR3lOSik9VNyqDj5a\ndex1wN3A5VXnWVp1nsFWvjHVULkF9bG/X32/VWRJktRGSgvIEdEH7AYsqDq0ANhzlKeuFxH3RMT9\nEXF+ROwyxmu8FfhWSmusLXZ4RCyOiN9FxKkRsf4Y850WEf2VDRh1vEaxTn/ZM5AkSRpRmRXk55C1\nQjxctf9hYNYIz/k9MA94PXAYsAy4MiK2GWH8G4CZwNlV+7+fP39v4N+BvwfOG2O+xwGDhe3+McZr\nJCOtamEVWZIktYGyWywAqiu7UWNfNjCl36SUvpdSujmldAXwJuAu4J9GOPe7gAtTSqulsZTS11NK\nl6SUbksp/RA4BHhVROw6yjxPBgYK2+ZjvTE1yJAsSZLaQJkBeTGwgjWrxZuwZlW5ppTSSuC3wBoV\n5Ih4HvAq4Bt1nOoG4Jla5ym81tMppaHKBjxRzxw1gpFWtZAkSSpZaQE5pTQMXA/sW3VoX+Cqes4R\nEQHsDDxY4/A7gUeAC+o41QuBtUc4jyaTNxCRJEklK3WZN+ALwHcj4jrgauC9wBzgLICI+A7w55TS\ncfnjTwK/Af4A9AMfIAvIRxdPGhFrkQXkc1JKy6uObQUcDvwfWRV7B+DzwI1kS81JkiSph5Xag5xS\n+hFwDNl6xDcBLwcOSCndkw+ZA2xWeMpM4GvAHWSrXTwXeHlK6dqqU78qf+63arzsMLAPcBFwJ/Cl\n/FyvSimtaMHbUr1Ga7OwH1mSJJUk1lz9TPXIl3obHBwcpL/fZcuaNrwkC8O1fOyBLERLkiQ1YWho\niIGBAYCB/BqyurTDKhbqZVaRJUlSmzEgS5IkSQUGZJWvcgtql32TJEltwICs9uEd9iRJUhswIEuS\nJEkFBmS1P6vIkiRpEhmQ1V5GWtXCO+xJkqRJYkCWJEmSCgzIaj+ujSxJkkpkQJYkSZIKDMhqT1aR\nJUlSSQzIkiRJUoEBWe1rtDvsWUWWJEkTxIAsSZIkFRiQ1f5GWxvZKrIkSWoxA7I6w2gX7UmSJLWQ\nAVmd7ZmlZc9AkiR1GQOyOketKvLntrbNQpIktZQBWZ3vpNlw/IBBWZIktYQBWZ3FG4hIkqQJZkCW\nJEmSCgzI6jyuaCFJkiaQAVndxTYLSZI0TgZkdabRbkMtSZI0DgZkdR+ryJIkaRwMyOps9iNLkqQW\nMyCrO1lFliRJTTIgq/ONVEUe9jbUkiSpcQZkdYdaIfnUrb3DniRJapgBWZIkSSowIKt7jNRqYT+y\nJElqgAFZ3aWyPvK/3Ln6fkOyJEmqkwFZ3Wna+mvuW7l88uchSZI6jgFZ3alWu8Upc6wiS5KkMRmQ\n1VtstZAkSWMwIKt7jXbRnsu/SZKkERiQ1d28FbUkSWqQAVndz+XfJElSAwzI6m2GZEmSVMWArN5Q\nWR/ZSrIkSRqDAVm9xXYLSZI0BgOyJEmSVGBAVu9x+TdJkjQKA7J602jLv9luIUlSTzMgS7UYkiVJ\n6lkGZPUubyIiSZJqMCCrt7n8myRJqmJAlsDl3yRJ0rMMyNJYXN1CkqSeYkCWKuxJliRJGJCl1bn8\nmyRJPW/cATki+iPiDRGxfSsmJEmSJJWp4YAcEf8VEe/Pv14XuA74L+CWiPj7Fs9PmnxWkSVJ6mnN\nVJBfDlyRf30wEMBM4APAJ1o0L6lcLv8mSVLPaiYgDwCP5V+/GviflNJS4AJgm1ZNTGprhmRJkrpW\nMwH5PuClETGDLCAvyPdvACxr1cSktuDKFpIk9ZypTTznNOD7wJPAPcBl+f6XA7e2ZlpSBzhp9qqv\nP/ZAFqYlSVLHa7iCnFL6CvBS4Ahgr5TSyvzQn7AHWd3IKrIkST2lqWXeUkrXpZR+nFJ6MiKmRMTO\nwFUppStbPD+pPYwVku1JliSpazSzzNtpEfGu/OspwOXADcB9EbF3a6cndRBDsiRJXaGZCvIhwM35\n1wcCzwdeQNab/OkWzUtqP/W0WhiSJUnqeM0E5OcAD+VfHwD8d0rpLuCbwE6tmpjUlkZbH1mSJHWF\nZgLyw8AOeXvFq4FL8v3TgRWtmpjU1sa6297xA1aSJUnqUM0E5G+T3Vr6NiABF+f7/xb4fYvmJXU+\n2y0kSepIzSzzdjzwbuBrwN+llJ7OD60ATmnd1KQ25/JvkiR1pUgplT2HjhQR/cDg4OAg/f39ZU9H\nZRpesvpNQ6p5ExFJkkoxNDTEwMAAwEBKaaje5zW1DnJEvCIifhYRf4yIP0TETyPiZc2cS5IkSWon\nzayD/FayC/OWAl8Cvgw8BVwaEW9p7fSkDuBNRCRJ6irNVJA/DnwkpXRoSulLKaXTU0qHAh8F/q21\n05O6hCFZkqSO0UxA3hL4WY39PyW7aUhDIuKoiFgYEcsi4vrRWjUiYl5EpBrbOoUxx9c4/lDVeSIf\n90BEPBURl0XECxudu/SsetZHdvk3SZI6QjMB+T5gnxr798mP1S0iDmXVHfh2Aa4ALoyIOaM8bQjY\nrLillJZVjfld1ZjqG5h8BDgWeD+wB9mNTy6OiPUbmb+0Ble2kCSp4zUTkD8PfCkivhoRb4uIt0bE\nWcDpwKkNnutY4JsppW+klO5IKR1DFrKPHOU5KaX0UHGrMWZ51ZhHKwciIoBjgE+nlM5LKd0GvIPs\nRif2UGv87EmWJKmjNbMO8leBN5NVZU8jC8Y7AoemlP6j3vNERB+wG7Cg6tACYM9RnrpeRNwTEfdH\nxPkRsUuNMdvk7RMLI+KHEbFl4djzgVnF183Xcr58tNeNiGkR0V/ZAKvNGpkhWZKkjtXUMm8ppR+n\nlPZKKW2Ub3sB/zdGa0S15wBTyG5dXfQwWYCt5ffAPOD1wGHAMuDKiNimMOYa4O3A/sB78nNdFREb\n5ccr527kdQGOAwYL2/2jjJUMyZIkdaimAvIIdgAWNvG86juVRI192cCUfpNS+l5K6eaU0hXAm4C7\ngH8qjLkwpfQ/KaVbU0qXAK/ND72j2dfNnQwMFLbNRxkrZca6eM+QLElS22llQG7UYrLbU1dXbTdh\nzepuTSmllcBvgW1GGbMEuLUwptKz3NDrppSeTikNVTbgiXrmKAFevCdJUgcpLSCnlIaB64F9qw7t\nC1xVzznyC+52Bh4cZcw0YPvCmIVkIXnfwpg+4BX1vq7UUi7/JklSWymzggzwBeDdEXFERGwfEV8E\n5gBnAUTEdyLi5MrgiPhkROwfEVtGxM7AN8kC8lmFMafmt8J+fkT8LXAu0A+cA9kSGGQXF34sIg6O\niB2Bs8nuDPiDSXjP6lX2JEuS1BGm1jswIl40xpDtGn3xlNKP8ovn5pOtV3wbcEBK6Z58yBxgZeEp\nM4GvkbVHDAI3Ai9PKV1bGLM58J9kFwE+CvwGeEnhnACfBdYFvgJsQHZh334pJdsmVK6TZmchum9G\n2TORJKlnRVZQrWNgxEqyi9iixuHK/pRSmtK66bWvfKm3wcHBQfr7+8uejjrJ8JIsCI/GkCxJ0rgN\nDQ0xMDAAMJBfQ1aXRgLy8+oZV1Wp7VoGZI2LIVmSpAk34QFZqzMgqyUMypIkTZhmA3LZF+lJGosX\n70mSNKkMyFKZ6l0f2ZAsSdKkMSBLZTMkS5LUVgzIUjvwTnuSJLUNA7LULuoJyVaRJUmacA2vYhER\nN5Kte1wtAcuAPwJnp5R+Of7ptS9XsdCEcnULSZLGbTJXsfg5sCWwBPglcBnwJLAV8FuyO+JdEhEH\nNXFuSWA1WZKkEjUTkJ8DfD6l9LKU0r+klI5NKb0cOBWYkVLaD/gU8G+tnKjUcwzJkiSVopmA/Cbg\nP2vs/2F+jPz4ds1OSpIkSSpLMwF5GbBnjf175scq53262UlJytVbRT5+wEqyJEktMrWJ55wBnBUR\nu5H1HCfgb4B3AyflY/YHbmzJDKVeVwnJY120VznuxXuSJI1Lw6tYAETE4cD7WdVGcSdwRkrpB/nx\ndYGUUlo2wik6nqtYqBSubiFJUt2aXcWimQoyKaXvA98f5fhTzZxXUgucNNuQLEnSODR9o5CI2C0i\n3hoRh0fELq2clKQReMc9SZImXMMBOSI2iYhfkPUffwn4MnB9RFwaERu3eoKSqrj8myRJE6qZCvIZ\nQD/wwpTShimlDYAd831fauXkJI3AkCxJ0oRpJiC/GjgypXRHZUdK6XbgaOA1rZqYpDEYkiVJmhDN\nBOS1gGdq7H+myfNJalbfDDh+cPSgbEiWJKkhzQTaXwCnR8Sza01FxHOBLwKXtmpikiRJUhmaCcjv\nB9YHFkXE3RHxR2Bhvu8DrZycpDqN1W7h3fYkSapbUzcKAYiIfYEXAAHcnlK6pJUTa3feKERtyRuJ\nSJL0rGZvFNJ0QF7jRBFbACeklI5oyQnbnAFZbcuQLEkS0HxAbuVFdRsC72jh+SQ1w9UtJEkaF1ed\nkHqVIVmSpJoMyFI38pbUkiQ1zYAsdat610h2dQtJklYztd6BEXHeGENmjnMukiZCpZo82oV7J832\nwj1JknKNVJAHx9juAb7T6glKagEv3JMkqW4tW+at17jMmzqSS8BJknpIOyzzJqndWUmWJGlMBmRJ\na/LiPUlSD7PFokm2WKgr2HIhSepitlhImhi2XEiSeowBWepl9d5QxJYLSVIPMSBLva6Ru+4ZlCVJ\nPcCALKnxW1PbdiFJ6mIGZEmZem5NXWRIliR1KQOypNU12nJhSJYkdRkDsqTxMSRLkrqMAVnSmhrt\nSZYkqYt4o5AmeaMQ9Yx6biZS4U1FJEltxBuFSCqf7RaSpC5gQJY0Ole3kCT1GAOypPq4uoUkqUcY\nkCXVz4v3JEk9wIv0muRFeuppjVy4B168J0kqhRfpSZo83ppaktTFDMiSmuPFe5KkLmVAljR5DMmS\npA5gQJY0Pl64J0nqMgZkSePnEnCSpC5iQJbUGo2G5OMHDMqSpLZkQJbUOq5uIUnqAq6D3CTXQZbG\n4FrJkqSSuQ6yJEmS1AIGZEkTw3WSJUkdyoAsqX148Z4kqQ0YkCVNrGbWSbaaLEkqkQFZ0sQzJEuS\nOoirWDTJVSykcWhkhQtXt5AkNclVLCR1JyvJkqRJZgW5SVaQpXFqdJ3kCivKkqQ6WUGW1Fma6UsG\nK8qSpAlnQJZUHkOyJKkN2WLRJFsspAng7aklSS1ki4WkztdoRdkbi0iSJoABWVJ7cc1kSVLJSg/I\nEXFURCyMiGURcX1EvGyUsfMiItXY1imMOS4ifhsRT0TEIxHxk4jYruo8l9U4xw8n8n1KakCzIdlq\nsiSpBUoNyBFxKHAa8GlgF+AK4MKImDPK04aAzYpbSmlZ4fgrgDOBlwD7AlOBBRFR3aj49arzvG/c\nb0hS6/TNgOMHrSZLkiZdqRfpRcQ1wA0ppSML++4AfpJSOq7G+HnAaSmlmQ28xsbAI8ArUkq/yvdd\nBtyUUjpmHHP3Ij1psrhmsiSpCR13kV5E9AG7AQuqDi0A9hzlqetFxD0RcX9EnB8Ru4zxUgP5v49V\n7T88IhZHxO8i4tSIWH+M+U6LiP7KBow6XlILuRycJGkSldli8RxgCvBw1f6HgVkjPOf3wDzg9cBh\nwDLgyojYptbgiAjgC8CvU0q3FQ59P3/+3sC/A38PnDfGfI8DBgvb/WOMl9RKhmRJ0iQprcUiImYD\nfwb2TCldXdj/ceBtKaUX1HGOtYAbgF+llD5Q4/iZwGuBvVJKIwbaiNgNuA7YLaV0wwhjpgHTCrvW\nB+63xUKaZLZbSJLq1HEtFsBiYAVrVos3Yc2qck0ppZXAb4E1KsgRcQZZpfn/jRaOczcAz9Q6T+G1\nnk4pDVU24Il65iipxbx4T5I0wUoLyCmlYeB6spUmivYFrqrnHHkLxc7Ag8V9EfFl4I3AK1NKC+s4\n1QuBtYvnkdTmxrMUnMvBSZJGUfY6yF8A3h0RR0TE9hHxRWAOcBZARHwnIk6uDI6IT0bE/hGxZUTs\nDHyTLCCfVTjnmcBbgbcAT0TErHxbNz/HVhExPyJ2j4i5EXEA8N/AjcCVk/CeJbVKs33JYEVZkjSi\nqWW+eErpRxGxETCfbC3i24ADUkr35EPmACsLT5kJfI2sLWOQLNS+PKV0bWFMZcm4y6pe7p3A2cAw\nsA/wQWA94D7gAuCElNKKlrwxSZOnEpKb6UuuPMf+ZElSQanrIHcy10GW2kyzF++BAVmSulSzF+kZ\nkJtkQJba1HiCMhiWJamLdOIqFpLUeuPpSwZ7kyVJBmRJXciQLEkaBwOyJNViSJaknmVAltSdmr2h\nSJEhWZJ6khfpNcmL9KQO48V7ktRzvEhPkkbTir5k78AnST3BgCypd4w3JINtF5LUAwzIknqLIVmS\nNAZ7kJtkD7LUBcbblwz2JktSG/NOepPMgCx1EYOyJHUlL9KTpGbZdiFJKjAgSxK4brIk6VkGZEkq\ncjk4Sep5BmRJqmY1WZJ6mgFZkkbSimqyIVmSOo4BWZJGM95qsiFZkjqOy7w1yWXepB7kcnCS1FFc\n5k2SJpq9yZLUE6wgN8kKsqRxV5StJkvShPJOepPMgCwJaE3bBRiWJWkC2GIhSWVoxV34wNYLSWoj\nVpCbZAVZUk22XUhS27CCLEntwLWTJanjGZAlqd0YkiWpVLZYNMkWC0mjatXFe2DbhSQ1yVUsJpkB\nWVLdvMGIJJXCHmRJaletWOnCtgtJmjQGZEmaDN6FT5I6hgFZkiZTK1a5OH7AoCxJE8iALEmTzWqy\nJLU1A7IklcVqsiS1JQOyJJXJC/gkqe24zFuTXOZNUsu5HJwktZTrIE8yA7KkCWNQlqSWcB1kSeoW\nXsQnSaWygtwkK8iSJo0VZUlqihVkSepWVpQlaVIZkCWpU4w3KBuSJakuBmRJ6jTjWRrOtZMlaUwG\nZEnqRN5kRJImjBfpNcmL9CS1DS/ik6SavEhPknqVd+OTpJYyIEtSNzAkS1LLGJAlqVu0KiTbmyyp\nx9mD3CR7kCV1hPH2J9ubLKmD2YMsSVpTK1a7sJosqccYkCWp2xmSJakhtlg0yRYLSR2r2bYL2y0k\ndZhmWywMyE0yIEvqaPYmS+oB9iBLkupn24UkjciALEm9ypAsSTXZYtEkWywkdR3bLiR1GVssJEnj\nY0VZkgADsiSpqBUh2TvxSepwBmRJ0upadctqQ7KkDmUPcpPsQZbU9cbbk1xkf7KkEtiDLElqrVZU\nkiusKEvqIFaQm2QFWVJPsZosqQN5J71JZkCW1JMMypI6iC0WkqSJ1+q2C1e8kNSGrCA3yQqyJNG6\nirLVZEkTwAqyJGnytaqi7EV8ktqIFeQmWUGWpIJW9iaDFWVJLWEFWZJUnlb2JoMVZUmlsoLcJCvI\nkjQKe5MltQEryJKk9mFvsqQOZkCWJE2MVoZkl4OTNIkMyJKkiePtqiV1oLYIyBFxVEQsjIhlEXF9\nRLxslLHzIiLV2NZp5JwRMS0izoiIxRGxJCJ+GhGbT9R7lKSe1TcDjh/MtvGGZavJkiZB6QE5Ig4F\nTgM+DewCXAFcGBFzRnnaELBZcUspLWvwnKcBBwNvBvYC1gPOj4gpLXprkqRqlbDcqqBsWJY0AUpf\nxSIirgFuSCkdWdh3B/CTlNJxNcbPA05LKc1s9pwRMQA8CrwtpfSj/Phs4D7ggJTSRXXM21UsJGk8\nWrl2sqtdSKqhI1exiIg+YDdgQdWhBcCeozx1vYi4JyLuj4jzI2KXBs+5G7B2cUxK6QHgtpFeN2/J\n6K9swPpjvkFJ0shsvZDUpspusXgOMAV4uGr/w8CsEZ7ze2Ae8HrgMGAZcGVEbNPAOWcBwymlvzbw\nuscBg4Xt/hHGSZIa1crWC0OypHEqOyBXVPd5RI192cCUfpNS+l5K6eaU/n979x51WV3Xcfz9EZyB\nZoYBL8jNQROlTHMK73dTLK1VXkpZ1UrL0pWpeUlxvOCULccLKCquTCNR1NTVhUQzyWsmaDKUgKig\nMSjIDMjA3JQZLr/+2PvAfg7P5TznOc8+t/drrb2eOXv/9jm/w4Y5n+fLd/9O+QrwLOBS4MX9PmeP\nYzYBaxubN/RJ0qANYiAL1ZQAABWLSURBVNUL+5MlLdGwA/KPgVu4Y9X2UO5YAZ5VKeVW4BtAp4Lc\ny3NuBVYkOaTX1y2l7C2l7OxswK5e5idJWqRBt14YkiUt0lADcillH7AZOL7r0PHAub08R5IA64Gr\nF/Gcm4GbmmOSHA48oNfXlSS1YJAVZYOypB7tP+wJAG8HzkxyPnAe8HxgHfBegCQfAq7qrGiR5A3A\n14DLgIOAl1AF5D/r9TlLKTuSnA6ckuQ6YDtwMnAR8LllfbeSpMXpVJSXuupF51xXvJC0gKEH5FLK\nx5PcFTiJak3ji6mWWruiHrIOuLVxysHA+6haKHYA/wM8tpTy34t4ToCXATcDnwAOBD4PPLeUcsvg\n36Ukack61eSlLg33piMMyZLmNfR1kMeV6yBL0pANYh1lg7I00cZyHWRJkvo2qP5ke5MldbGC3Ccr\nyJI0Qgb1rXxWlKWJ0m8F2YDcJwOyJI0gg7KkBgNyywzIkjTCBhWUwbAsjTF7kCVJ6hjUV1eDfcrS\nFDIgS5Im16CCsiFZmiq2WPTJFgtJGkMuDSdNFVssJElaiEvDSeqBFeQ+WUGWpAlgRVmaaK5i0TID\nsiRNEIOyNJFssZAkqV+2XkhqsILcJyvIkjSBXD9Zmii2WLTMgCxJE85v5ZPGngG5ZQZkSZoCg6wo\ng2FZapk9yJIkDdogv5EP7FOWxoQBWZKkhQziJr6ONx0BG9calKURZotFn2yxkKQpZuuFNBZssZAk\nqS2DrCiDrRfSiDEgS5LUj+UIybZeSCPBFos+2WIhSZph0G0XYOuFtEQu89YyA7IkaU5+4Yg0EuxB\nliRpVLjqhTTWrCD3yQqyJKlnfiufNBS2WLTMgCxJWjRbL6RW2WIhSdKoG3TrhW0X0rKwgtwnK8iS\npCWz9UJaVrZYtMyALEkaGIOytCwMyC0zIEuSlsUgwrJBWQLsQZYkaTIMok/Z/mRpSQzIkiSNmhWr\nYOOOpQVlQ7LUN1ss+mSLhSSpVUttvbDtQlPIFgtJkibZUlsvrChLPbOC3CcryJKkobGaLPXECrIk\nSdNiENXkjWutKEtzsILcJyvIkqSRspSqshVlTSgryJIkTbOlVJWtKEszWEHukxVkSdJI8lv5pNv4\nTXotMyBLkkaeN/NpytliIUmSZvJmPqkvVpD7ZAVZkjQ2BtV2AVaVNVZssWiZAVmSNLZsvdCUsMVC\nkiT1ZsUq2Lhjaate2HahCWYFuU9WkCVJE8WqsiaQFWRJktQ/q8rSbQzIkiTpdn7hiGSLRb9ssZAk\nTYV+Wy9sudAIcBWLlhmQJUlTZSk9yoZlDYkBuWUGZEnSVDIoa4wYkFtmQJYkTT3Dskacq1hIkqR2\neUOfJpQBWZIk9c/l4TSBDMiSJGnpllpNNiRrhNiD3Cd7kCVJWkA/Pcr2JmuAvEmvZQZkSZJ6ZFDW\nkBiQW2ZAliSpD4sNywZlLYEBuWUGZEmSlsCgrBYYkFtmQJYkaUAWE5YNyloE10GWJEnjaTFLxbni\nhVpgQJYkSaOh16BsSNYys8WiT7ZYSJK0zHptvbDtQnOwB7llBmRJklpiUFafDMgtMyBLktQyb+bT\nIhmQW2ZAliRpiKwqqweuYiFJkqaHN/RpGVlB7pMVZEmSRkwvVWUrylPFCrIkSZpuvVSVrSirBwZk\nSZI0WVasWjgkb1xrUNacbLHoky0WkiSNib27YNNRsx878Qo48OB256PWuIpFywzIkiSNmb27YdOR\ncx+3P3nijHUPcpIXJrk8yY1JNid5TI/nnZCkJDmra3+ZY3tlY8yWWY6/edDvTZIkjYiVqxduvdi7\nq735aGQNvYKc5NnAmcALga8CLwD+GLh/KeUH85x3dD3+/4DtpZSnNY4d1jX8KcDpwDGllP+rx2yp\n972/MW53KWV3j/O2gixJ0jjbfS2cfMzsx07cAgce0up0NHhj22KR5OvABaWUP23s+zZwVillwxzn\n7Ad8GfgA8Bjg4GZAnmX8WcCaUsoTG/u2AKeWUk7tc94GZEmSJsGurXDKsXMft/VibI1li0WSFcBx\nwDldh84BHjnPqScB15ZSTu/hNe4B/DpVtbjbiUmuS/K/SV5bz2eu51mZ5KDOBqxZ6LUlSdIYWHOY\nS8Nphv2H/Pp3A/YDtnXt3wZ0t0kAkORRwPOA9T2+xnOAXcA/d+1/J3ABcD3wUGATcG+q9o7ZbADe\n0ONrSpKkcdJZQxngpzfAW46eebzzBSRWk6fCsANyR3efR2bZR5I1wIeBPyml/LjH5/4j4COllBtn\nvGAp72g8vDDJ9cA/JjmxlHLdLM+zCXh74/Ea4Moe5yBJksbFgQdXQXi2b+Vr7jMsT6xhB+QfA7dw\nx2rxodyxqgxwH+BewNlJOvvuBJDkZuDYUsr3Owfq1TCOBZ7dw1y+Vv88BrhDQC6l7AX2Np67h6eU\nJEljqVNRnu/rq990hCF5Qg21B7mUsg/YDBzfdeh44NxZTvkO8ECq9orO9kngi/Wff9g1/nnA5lLK\nN3uYzi/VP6/uafKSJGny9fKtfNu+1d581IphV5Chals4M8n5wHnA84F1wHsBknwIuKqUsqFuk7i4\neXKSGwBKKd37DwJ+B3hF9wsmeQTwcKpgvQN4CPAO4JPzLS0nSZKm0ELV5L9prCvg8nATYegBuZTy\n8SR3pVqZ4nCqAPzUUsoV9ZB1wK19PPUJVL3M/zDLsb1UbRdvAFYCV1Cth/zWPl5HkiRNg17aLt5y\nr+rnK78Hq+7e2tQ0WENfB3lcuQ6yJElTbr6g3PHizXDXOb6MRMtubL8oZFwZkCVJ0gx+4cjIMSC3\nzIAsSZJmtXcXbDpq7uOv/iEcYHZogwG5ZQZkSZK0oMu/Ah/8jdmP/flFcMi6duczZQzILTMgS5Kk\nnm3fAu960NzHN1wFK1e3Np1pYUBumQFZkiQt2kJ9yq/+ARywtr35TDgDcssMyJIkqW97d8OmI+cf\n4019S9ZvQB7qN+lJkiRNpZWrqzWVF/qWvn172puTbmMFuU9WkCVJ0sAstKay1eS+2GLRMgOyJEka\nuF6+fMSw3DNbLCRJksZd5+usN1w19xhbL5adFeQ+WUGWJEnLzorykthi0TIDsiRJao1BuS8G5JYZ\nkCVJUusMyotiD7IkSdKkW7Fq/qXhoArQe65tZz4Tygpyn6wgS5KkoeulogxTW1W2xaJlBmRJkjQy\nDMqzssVCkiRpWnWWh+ul/cIl4hZkBblPVpAlSdJI27sbNh05/5gJryhbQZYkSdLtVq7uraK8ca1V\n5S5WkPtkBVmSJI2NXnuUYaKqyt6k1zIDsiRJGjtTFpQNyC0zIEuSpLE2BStfGJBbZkCWJEkTY8+1\n8LZj5h8zhkHZm/QkSZLUn1V394a+BivIfbKCLEmSJtIEtV7YYtEyA7IkSZpYi7mZD0Y2LBuQW2ZA\nliRJU2GMK8r2IEuSJGnwVqxauD8ZJqpH2Qpyn6wgS5KkqdVLVXkEKsq2WLTMgCxJkqbeiAdlA3LL\nDMiSJEm1XoLyhqtg5ep25lMzILfMgCxJkjSLvbtg01Hzj2mpquxNepIkSRq+lWtg4475b+y79Zb2\n5tOH/Yc9AUmSJE2gFauqoAywaxuccr/bj91pv+HMqUcGZEmSJC2vNfeownIpcPONcOcDhz2jedli\nIUmSpHYkIx+OwYAsSZIkzWBAliRJkhoMyJIkSVKDAVmSJElqMCBLkiRJDQZkSZIkqcGALEmSJDUY\nkCVJkqQGA7IkSZLUYECWJEmSGgzIkiRJUoMBWZIkSWowIEuSJEkNBmRJkiSpwYAsSZIkNRiQJUmS\npAYDsiRJktRgQJYkSZIaDMiSJElSgwFZkiRJajAgS5IkSQ0GZEmSJKnBgCxJkiQ1GJAlSZKkBgOy\nJEmS1LD/sCcw7nbu3DnsKUiSJGkW/ea0lFIGPJXpkORI4Mphz0OSJEkLOqqUclWvgw3IfUoS4Ahg\n17DnMqHWUP0CchT+M55EXt/J5bWdbF7fyTap13cN8KOyiNBri0Wf6n/IPf8mosWpfv8AYFcpxT6W\nCeP1nVxe28nm9Z1sE3x9F/1evElPkiRJajAgS5IkSQ0GZI2qvcBf1j81eby+k8trO9m8vpPN61vz\nJj1JkiSpwQqyJEmS1GBAliRJkhoMyJIkSVKDAVmSJElqMCCrNUkem+TsJD9KUpI8ret4kmysj/80\nyZeS/ELXmEOSnJlkR72dmeTgdt+JuiXZkOQbSXYluSbJWUmO7RqzMsm7k/w4yZ4kn0xyVNeYdfW/\nI3vqce9KsqLdd6NuSf40yYVJdtbbeUme0jjutZ0Q9X/LJcmpjX1e3zFVf6aWrm1r47ifu3MwIKtN\nq4BvAi+a4/irgJfXxx8CbAX+I8maxpiPAuuBX6u39cCZyzVh9exxwHuAhwPHU31L5zlJVjXGnAo8\nHTgBeDSwGvhUkv0A6p+fpvr35NH1uGcCp7T0HjS3K4FXAw+uty8A/9r4IPXaToAkDwGeD1zYdcjr\nO96+BRze2B7YOObn7lxKKW5urW9AAZ7WeBzgauDExr6VwA3AC+rHP1+f97DGmIfX+44d9ntym3F9\n715fl8fWj9cC+4BnN8YcAdwC/Gr9+Cn14yMaY04AbgQOGvZ7crvDNd4OPM9rOxkbVei9FHgS8CXg\n1Hq/13eMN2Aj8L9zHPNzd57NCrJGxb2Bw4BzOjtKKXuBLwOPrHc9AthRSvl6Y8zXgB2NMRoNa+uf\n2+ufxwF3Zub1/RFwMTOv78X1/o7PUv2FfdyyzlY9S7JfkhOoqoXn4bWdFO8BPl1K+VzXfq/v+Ltv\n3UJxeZKPJfnZer+fu/PYf9gTkGqH1T+3de3fBhzdGHPNLOde0zhfQ5YkwNuB/yqlXFzvPgzYV0q5\nvmv4Nm6/dofRdf1LKdcn2YfXd+iSPJAqEB8A7AaeXkq5JMl6vLZjrf6F55ep/hd7N//bHW9fB/6A\n6v8O3AN4HXBu3R7l5+48DMgaNd1f7ZiufbN99WP3GA3XacAvUvUiLsTrOz6+S9V7eDBVf+kHkzxu\nnvFe2zGQ5J7AO4Enl1JuXMypeH1HXinlM42HFyU5D/g+8Bzga51hXad5bfEmPY2Ozl213b+RHsrt\nv91upfoNuNvdueNvwBqCJO8GfhN4QinlysahrcCKJId0ndJ9fWdc/3r8nfH6Dl0pZV8p5XullPNL\nKRuobrj9c7y24+44qmu1OcnNSW6muun2JfWft+H1nRillD3ARcB98XN3XgZkjYrLqf5DPL6zo14i\n6HHAufWu84C1SR7aGPMwqn7Xc9HQ1EsFnQY8A/iVUsrlXUM2Azcx8/oeDjyAmdf3AfX+jicDe+vz\nNVpC1WPqtR1vn6da1WB9Yzsf+Ejjz17fCZFkJdWNd1fj5+68bLFQa5KsBo5p7Lp33b+4vZTyg3rd\nzdckuQy4DHgN8BOqJWYopXw7yb8D70/ygvo53gd8qpTy3dbeiGbzHuB3gd8CdiXpVCR2lFJ+WkrZ\nkeR04JQk11HdvHcyVSWjc1PQOcAlwJlJXgncpR7z/lLKzhbfi7okeRPwGeCHwBqqFQoeD/ya13a8\nlVJ2Ud1wd5ske4DrOvcQeH3HV5KTgbOBH1BVhl8HHAR8sJRS/Nydx7CX0XCbno3qA7XMsp1RHw/V\nkjRXUy0P9GXgAV3PcRfgw8DOevswcPCw39u0b3Nc1wI8tzHmAODdwHVUfwGfDdyz63nWAZ+qj19X\nj1857Pc37RtwOrCFqiJ4DVUwOt5rO5kbjWXevL7jvQEfA35EtVTfVcA/AfdvHPdzd44t9ZuXJEmS\nhD3IkiRJ0gwGZEmSJKnBgCxJkiQ1GJAlSZKkBgOyJEmS1GBAliRJkhoMyJIkSVKDAVmSJElqMCBL\n0hRJsiXJS4c9D0kaZQZkSZpASZ6b5IZZDj0EeF8Lr28QlzS29h/2BCRJ7SmlXDvsOSxGkhWllH3D\nnoek6WIFWZKWUZIvJXlXkrcm2Z5ka5KNPZ67Nsn7klyTZGeSLyR5UOP4g5J8Mcmu+vjmJA9O8njg\nA8DaJKXeNtbnzKjs1sdekORTSX6S5NtJHpHkmHrue5Kcl+Q+jXPuk+Rfk2xLsjvJN5I8qfmegaOB\nd3Rev3HsmUm+lWRvPZdXdL3nLUlel+SMJDuA9ydZkeS0JFcnubEes2FRF0KSFsGALEnL7znAHuBh\nwKuAk5IcP98JSQJ8GjgMeCpwHHAB8Pkkd6mHfQS4kqpt4jjgzcBNwLnAS4GdwOH1dvI8L/d64EPA\neuA7wEeBvwU2AQ+ux5zWGL8a+DfgScAvAZ8Fzk6yrj7+jHpeJzVenyTHAZ8APgY8ENgIvDHJc7vm\n80rg4vo9vRF4CfCbwLOAY4HfB7bM834kaUlssZCk5XdhKeUv6z9fluRFwBOB/5jnnCdQhchDSyl7\n631/keRpwG9T9RGvA95WSvlO57k7J9fV11JK2drD/D5QSvlEfd5bgPOAN5ZSPlvveydVRRqqJ/0m\n8M3G+a9L8nSqEHtaKWV7kluAXV2v/3Lg86WUN9aPL01yf6pAfEZj3BdKKbcF+jp4Xwb8VymlAFf0\n8J4kqW9WkCVp+V3Y9fhq4NAFzjmOqlJ7Xd3GsDvJbuDeQKfd4e3A3yX5XJJXN9sgljC/bfXPi7r2\nHZDkIIAkq+qWkUuS3FDP6+eoAvt8fh74ate+rwL3TbJfY9/5XWPOoKpuf7duV3nygu9IkpbAgCxJ\ny++mrseFhf/+vRNVkF7ftR0LvA2glLIR+AWqVoxfAS6pK7lLmV+ZZ19nzm8Dngm8FnhMPa+LgBUL\nvE4az9Xc121P80Ep5QKqXwxeDxwIfCLJPy7wWpLUN1ssJGk0XUDVf3xzKWXLXINKKZcCl1LdEPcP\nwB8C/wLsA/ab67wlegxwRinlXwCSrAbu1TVmtte/BHh0175HApeWUm6Z7wVLKTuBjwMfr8Pxvye5\nSylle39vQZLmZgVZkkbT56h6gc9K8qtJ7pXkkUn+ul6p4sB6ZYfHJzk6yaOobtb7dn3+FmB1kicm\nuVuSnxng3L4HPCPJ+npVjY9yx8+TLcBjkxyZ5G71vlOAJyZ5fZL7JXkO8CLmv4GQJC9LckKSn0ty\nP+B3gK3AbOs8S9KSGZAlaQTVN6M9FfhP4O+pqsQfo6rUbgNuAe5KtfrEpVSrQ3wGeEN9/rnAe6mq\nrtdSrZ4xKC8DrqdaLeNsqlUsLugac1I91+/Xr99plXgWcALVKhV/BZxUSjljgdfbDZxI1Zv8jfp5\nn1pKuXXJ70SSZpHq72BJkiRJYAVZkiRJmsGALElDkOT3msu3dW3fGvb8JGma2WIhSUOQZA1wjzkO\n31RK8cswJGlIDMiSJElSgy0WkiRJUoMBWZIkSWowIEuSJEkNBmRJkiSpwYAsSZIkNRiQJUmSpAYD\nsiRJktTw/9ZzfQnelRspAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x113332e80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 绘图, 放大细节查看拐点后 logloss 变化趋势\n",
    "cv_result = pd.DataFrame.from_csv('1_nestimators.csv')\n",
    "cv_result = cv_result.iloc[50:]\n",
    "test_means = cv_result['test-mlogloss-mean']\n",
    "test_stds = cv_result['test-mlogloss-std']\n",
    "train_means = cv_result['train-mlogloss-mean']\n",
    "train_stds = cv_result['train-mlogloss-std']\n",
    "x_axis = range(50, cv_result.shape[0]+50)\n",
    "plt.figure(figsize=(8,8), dpi=100)\n",
    "plt.errorbar(x_axis, test_means, yerr=test_stds, label='test')\n",
    "plt.errorbar(x_axis, train_means, yerr=train_stds, label='train')\n",
    "plt.title('Log Loss vs XGBoost n_estimators')\n",
    "plt.xlabel('n_estimators')\n",
    "plt.ylabel('Log Loss')\n",
    "plt.legend(loc='best')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其实从图上看, 在 n_estimators=300 处 log 损失减小已经很慢了."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step II: 调整树的参数 max_depth & min_child_weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第一轮参数整定得到 n_estimators 最优值 522, 其余参数继续保持原值."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': [6, 7, 8], 'min_child_weight': [3, 4]}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# max_depth 从 6 开始, 逐步加大到 10; min_child_weight = 1／sqrt(ratio_rare_event) = 3.6, 在 3.6 附近取值[3, 4]\n",
    "max_depth = [6, 7, 8]\n",
    "min_child_weight = [3, 4]\n",
    "param_test2_1 = dict(max_depth=max_depth, min_child_weight=min_child_weight)\n",
    "param_test2_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "xgb2_1 = XGBClassifier(n_jobs=8, \n",
    "                     learning_rate=0.1, \n",
    "                     n_estimators=522, # 第 1 步得到\n",
    "                     max_depth=5, \n",
    "                     min_child_weight=1, \n",
    "                     gamma=0, \n",
    "                     subsample=1, \n",
    "                     colsample_bytree=0.4, \n",
    "                     colsample_bylevel=0.4, \n",
    "                     objective='multi:softprob', \n",
    "                     seed=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.58014, std: 0.00341, params: {'max_depth': 6, 'min_child_weight': 3},\n",
       "  mean: -0.58021, std: 0.00345, params: {'max_depth': 6, 'min_child_weight': 4},\n",
       "  mean: -0.58354, std: 0.00368, params: {'max_depth': 7, 'min_child_weight': 3},\n",
       "  mean: -0.58302, std: 0.00422, params: {'max_depth': 7, 'min_child_weight': 4},\n",
       "  mean: -0.58889, std: 0.00476, params: {'max_depth': 8, 'min_child_weight': 3},\n",
       "  mean: -0.58786, std: 0.00453, params: {'max_depth': 8, 'min_child_weight': 4}],\n",
       " {'max_depth': 6, 'min_child_weight': 3},\n",
       " -0.58014408810704121)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch2_1 = GridSearchCV(xgb2_1, param_grid=param_test2_1, scoring='neg_log_loss', n_jobs=8, cv=kfold)\n",
    "gsearch2_1.fit(X_train, y_train)\n",
    "gsearch2_1.grid_scores_, gsearch2_1.best_params_, gsearch2_1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([ 281.90768318,  281.41305542,  312.65043154,  317.04650178,\n",
       "         341.68148403,  308.57801733]),\n",
       " 'mean_score_time': array([ 4.89741206,  3.88110056,  5.05291986,  3.09118819,  4.99946022,\n",
       "         2.62805099]),\n",
       " 'mean_test_score': array([-0.58014409, -0.58020817, -0.5835361 , -0.58301621, -0.58888652,\n",
       "        -0.58786256]),\n",
       " 'mean_train_score': array([-0.42222601, -0.42506661, -0.36187922, -0.36645974, -0.29610284,\n",
       "        -0.30427724]),\n",
       " 'param_max_depth': masked_array(data = [6 6 7 7 8 8],\n",
       "              mask = [False False False False False False],\n",
       "        fill_value = ?),\n",
       " 'param_min_child_weight': masked_array(data = [3 4 3 4 3 4],\n",
       "              mask = [False False False False False False],\n",
       "        fill_value = ?),\n",
       " 'params': [{'max_depth': 6, 'min_child_weight': 3},\n",
       "  {'max_depth': 6, 'min_child_weight': 4},\n",
       "  {'max_depth': 7, 'min_child_weight': 3},\n",
       "  {'max_depth': 7, 'min_child_weight': 4},\n",
       "  {'max_depth': 8, 'min_child_weight': 3},\n",
       "  {'max_depth': 8, 'min_child_weight': 4}],\n",
       " 'rank_test_score': array([1, 2, 4, 3, 6, 5], dtype=int32),\n",
       " 'split0_test_score': array([-0.57486381, -0.57448359, -0.57773462, -0.57689954, -0.58132746,\n",
       "        -0.5812666 ]),\n",
       " 'split0_train_score': array([-0.42457925, -0.42580859, -0.36295573, -0.36916545, -0.30012106,\n",
       "        -0.30748116]),\n",
       " 'split1_test_score': array([-0.57846907, -0.57904612, -0.58230031, -0.58113446, -0.58802643,\n",
       "        -0.58719971]),\n",
       " 'split1_train_score': array([-0.42318013, -0.42507702, -0.36412544, -0.36960577, -0.29795365,\n",
       "        -0.30392253]),\n",
       " 'split2_test_score': array([-0.57992715, -0.57999756, -0.58284455, -0.5817276 , -0.58716112,\n",
       "        -0.58565478]),\n",
       " 'split2_train_score': array([-0.42054852, -0.42445703, -0.36009431, -0.36351288, -0.29192629,\n",
       "        -0.30296941]),\n",
       " 'split3_test_score': array([-0.58281474, -0.58391481, -0.58635986, -0.58639171, -0.59321939,\n",
       "        -0.59046954]),\n",
       " 'split3_train_score': array([-0.4210824 , -0.424784  , -0.36193877, -0.36377224, -0.29402207,\n",
       "        -0.30200143]),\n",
       " 'split4_test_score': array([-0.58464703, -0.58359981, -0.58844264, -0.58892954, -0.59469998,\n",
       "        -0.59472424]),\n",
       " 'split4_train_score': array([-0.42173976, -0.4252064 , -0.36028185, -0.36624237, -0.29649111,\n",
       "        -0.30501169]),\n",
       " 'std_fit_time': array([ 0.20253021,  0.72594652,  0.28295165,  4.1594146 ,  9.89041609,\n",
       "         9.63839465]),\n",
       " 'std_score_time': array([ 0.97193411,  1.90680947,  0.62949975,  1.17928754,  1.15492355,\n",
       "         0.57679611]),\n",
       " 'std_test_score': array([ 0.00341   ,  0.00344697,  0.00367987,  0.00421832,  0.00476283,\n",
       "         0.00453192]),\n",
       " 'std_train_score': array([ 0.00147013,  0.00045203,  0.00154567,  0.00257575,  0.00288082,\n",
       "         0.00188786])}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch2_1.cv_results_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最佳参数'max_depth': 6, 'min_child_weight': 3 都在范围边缘取得, 尝试继续缩小范围."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': [4, 5, 6], 'min_child_weight': [1, 2, 3]}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_depth = [4, 5, 6]\n",
    "min_child_weight = [1, 2, 3]\n",
    "param_test2_1 = dict(max_depth=max_depth, min_child_weight=min_child_weight)\n",
    "param_test2_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.58525, std: 0.00386, params: {'max_depth': 4, 'min_child_weight': 1},\n",
       "  mean: -0.58547, std: 0.00377, params: {'max_depth': 4, 'min_child_weight': 2},\n",
       "  mean: -0.58534, std: 0.00369, params: {'max_depth': 4, 'min_child_weight': 3},\n",
       "  mean: -0.58171, std: 0.00433, params: {'max_depth': 5, 'min_child_weight': 1},\n",
       "  mean: -0.58200, std: 0.00442, params: {'max_depth': 5, 'min_child_weight': 2},\n",
       "  mean: -0.58250, std: 0.00393, params: {'max_depth': 5, 'min_child_weight': 3},\n",
       "  mean: -0.58057, std: 0.00356, params: {'max_depth': 6, 'min_child_weight': 1},\n",
       "  mean: -0.58021, std: 0.00426, params: {'max_depth': 6, 'min_child_weight': 2},\n",
       "  mean: -0.58014, std: 0.00341, params: {'max_depth': 6, 'min_child_weight': 3}],\n",
       " {'max_depth': 6, 'min_child_weight': 3},\n",
       " -0.58014408810704121)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb2_1 = XGBClassifier(n_jobs=8, \n",
    "                     learning_rate=0.1, \n",
    "                     n_estimators=522, \n",
    "                     max_depth=5, \n",
    "                     min_child_weight=1, \n",
    "                     gamma=0, \n",
    "                     subsample=1, \n",
    "                     colsample_bytree=0.4, \n",
    "                     colsample_bylevel=0.4, \n",
    "                     objective='multi:softprob', \n",
    "                     seed=3)\n",
    "gsearch2_1 = GridSearchCV(xgb2_1, param_grid=param_test2_1, scoring='neg_log_loss', n_jobs=8, cv=kfold)\n",
    "gsearch2_1.fit(X_train, y_train)\n",
    "gsearch2_1.grid_scores_, gsearch2_1.best_params_, gsearch2_1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([ 217.72007775,  218.01974149,  217.84971852,  249.27891979,\n",
       "         249.74728332,  248.55005736,  281.72564988,  281.02238731,\n",
       "         232.75401545]),\n",
       " 'mean_score_time': array([ 1.43549423,  1.61257219,  1.48421769,  2.02771921,  2.33315511,\n",
       "         2.14243512,  2.26723514,  2.77339168,  2.07663555]),\n",
       " 'mean_test_score': array([-0.58524684, -0.58547195, -0.58533587, -0.5817075 , -0.58199531,\n",
       "        -0.58249999, -0.58057451, -0.58020671, -0.58014409]),\n",
       " 'mean_train_score': array([-0.52028802, -0.52173233, -0.52224336, -0.47449418, -0.47617774,\n",
       "        -0.4773601 , -0.4156003 , -0.41984114, -0.42222601]),\n",
       " 'param_max_depth': masked_array(data = [4 4 4 5 5 5 6 6 6],\n",
       "              mask = [False False False False False False False False False],\n",
       "        fill_value = ?),\n",
       " 'param_min_child_weight': masked_array(data = [1 2 3 1 2 3 1 2 3],\n",
       "              mask = [False False False False False False False False False],\n",
       "        fill_value = ?),\n",
       " 'params': [{'max_depth': 4, 'min_child_weight': 1},\n",
       "  {'max_depth': 4, 'min_child_weight': 2},\n",
       "  {'max_depth': 4, 'min_child_weight': 3},\n",
       "  {'max_depth': 5, 'min_child_weight': 1},\n",
       "  {'max_depth': 5, 'min_child_weight': 2},\n",
       "  {'max_depth': 5, 'min_child_weight': 3},\n",
       "  {'max_depth': 6, 'min_child_weight': 1},\n",
       "  {'max_depth': 6, 'min_child_weight': 2},\n",
       "  {'max_depth': 6, 'min_child_weight': 3}],\n",
       " 'rank_test_score': array([7, 9, 8, 4, 5, 6, 3, 2, 1], dtype=int32),\n",
       " 'split0_test_score': array([-0.5790543 , -0.57948316, -0.57952875, -0.57437592, -0.57511978,\n",
       "        -0.57638621, -0.5754199 , -0.57284494, -0.57486381]),\n",
       " 'split0_train_score': array([-0.52172739, -0.52309688, -0.5243828 , -0.47611857, -0.47820615,\n",
       "        -0.47891054, -0.41634175, -0.42134306, -0.42457925]),\n",
       " 'split1_test_score': array([-0.58380945, -0.5841173 , -0.58350138, -0.58088272, -0.58046074,\n",
       "        -0.58162159, -0.57963161, -0.58009631, -0.57846907]),\n",
       " 'split1_train_score': array([-0.52033449, -0.52179401, -0.52199175, -0.47479798, -0.47506961,\n",
       "        -0.47693459, -0.41647136, -0.41949085, -0.42318013]),\n",
       " 'split2_test_score': array([-0.58478689, -0.58470809, -0.58511596, -0.58120402, -0.58082993,\n",
       "        -0.58095364, -0.57892601, -0.57922972, -0.57992715]),\n",
       " 'split2_train_score': array([-0.51986287, -0.52184015, -0.52271761, -0.47326543, -0.47472049,\n",
       "        -0.47790502, -0.414268  , -0.41917481, -0.42054852]),\n",
       " 'split3_test_score': array([-0.58857844, -0.58947967, -0.58947077, -0.58512541, -0.58608594,\n",
       "        -0.58636584, -0.58318395, -0.5842649 , -0.58281474]),\n",
       " 'split3_train_score': array([-0.51970032, -0.52151509, -0.52119091, -0.47488116, -0.47751962,\n",
       "        -0.4770342 , -0.41506257, -0.42026337, -0.4210824 ]),\n",
       " 'split4_test_score': array([-0.59000656, -0.58957278, -0.58906364, -0.58695104, -0.58748182,\n",
       "        -0.58717411, -0.58571263, -0.584599  , -0.58464703]),\n",
       " 'split4_train_score': array([-0.51981501, -0.52041553, -0.52093374, -0.47340775, -0.47537283,\n",
       "        -0.47601612, -0.41585783, -0.4189336 , -0.42173976]),\n",
       " 'std_fit_time': array([ 0.21771689,  0.25898868,  0.25978133,  1.17958281,  2.1848744 ,\n",
       "         0.25876299,  1.23863852,  1.29880073,  1.07487647]),\n",
       " 'std_score_time': array([ 0.42927186,  0.56070237,  0.25633084,  0.9356973 ,  0.99130974,\n",
       "         0.24116037,  1.04309015,  1.21346143,  0.51625901]),\n",
       " 'std_test_score': array([ 0.00385687,  0.00377258,  0.00369134,  0.00433267,  0.00442307,\n",
       "         0.00393272,  0.00356039,  0.00426434,  0.00341   ]),\n",
       " 'std_train_score': array([ 0.00075151,  0.0008554 ,  0.00124001,  0.00105554,  0.00140815,\n",
       "         0.00097922,  0.00082934,  0.00087457,  0.00147013])}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch2_1.cv_results_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以得出 max_depth 最优值为 6, min_child_weight 最优值为 3. 当前最高得分: -0.58014408810704121."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step III: 调整树的列采样参数: colsample_bytree 和 colsample_bylevel(行采样参数 subsample 保持不变为 1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "根据 Owen Zhang 看法, 两个参数调整范围都为 [0.3, 0.5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'colsample_bylevel': [0.3, 0.35, 0.4, 0.45, 0.5],\n",
       " 'colsample_bytree': [0.3, 0.35, 0.4, 0.45, 0.5]}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "colsample_bytree = [0.3, 0.35, 0.4, 0.45, 0.5]\n",
    "colsample_bylevel = [0.3, 0.35, 0.4, 0.45, 0.5]\n",
    "param_test_3_1 = dict(colsample_bytree=colsample_bytree, colsample_bylevel=colsample_bylevel)\n",
    "param_test_3_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "采用前两步得到的最优参数: n_estimators=522, max_depth=6, min_child_weight=3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "xgb3_1 = XGBClassifier(n_jobs=7, \n",
    "                     learning_rate=0.1, \n",
    "                     n_estimators=522, # 第 1 步得到\n",
    "                     max_depth=6, # 第 2 步得到\n",
    "                     min_child_weight=3, # 第 2 步得到\n",
    "                     gamma=0, \n",
    "                     subsample=1, \n",
    "                     colsample_bytree=0.4, \n",
    "                     colsample_bylevel=0.4, \n",
    "                     objective='multi:softprob', \n",
    "                     seed=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.58087, std: 0.00388, params: {'colsample_bylevel': 0.3, 'colsample_bytree': 0.3},\n",
       "  mean: -0.58055, std: 0.00437, params: {'colsample_bylevel': 0.3, 'colsample_bytree': 0.35},\n",
       "  mean: -0.58105, std: 0.00327, params: {'colsample_bylevel': 0.3, 'colsample_bytree': 0.4},\n",
       "  mean: -0.58045, std: 0.00482, params: {'colsample_bylevel': 0.3, 'colsample_bytree': 0.45},\n",
       "  mean: -0.58028, std: 0.00388, params: {'colsample_bylevel': 0.3, 'colsample_bytree': 0.5},\n",
       "  mean: -0.58026, std: 0.00361, params: {'colsample_bylevel': 0.35, 'colsample_bytree': 0.3},\n",
       "  mean: -0.58082, std: 0.00435, params: {'colsample_bylevel': 0.35, 'colsample_bytree': 0.35},\n",
       "  mean: -0.58050, std: 0.00316, params: {'colsample_bylevel': 0.35, 'colsample_bytree': 0.4},\n",
       "  mean: -0.58112, std: 0.00393, params: {'colsample_bylevel': 0.35, 'colsample_bytree': 0.45},\n",
       "  mean: -0.58135, std: 0.00394, params: {'colsample_bylevel': 0.35, 'colsample_bytree': 0.5},\n",
       "  mean: -0.58045, std: 0.00430, params: {'colsample_bylevel': 0.4, 'colsample_bytree': 0.3},\n",
       "  mean: -0.58075, std: 0.00359, params: {'colsample_bylevel': 0.4, 'colsample_bytree': 0.35},\n",
       "  mean: -0.58014, std: 0.00341, params: {'colsample_bylevel': 0.4, 'colsample_bytree': 0.4},\n",
       "  mean: -0.58044, std: 0.00393, params: {'colsample_bylevel': 0.4, 'colsample_bytree': 0.45},\n",
       "  mean: -0.58126, std: 0.00371, params: {'colsample_bylevel': 0.4, 'colsample_bytree': 0.5},\n",
       "  mean: -0.58120, std: 0.00416, params: {'colsample_bylevel': 0.45, 'colsample_bytree': 0.3},\n",
       "  mean: -0.58065, std: 0.00382, params: {'colsample_bylevel': 0.45, 'colsample_bytree': 0.35},\n",
       "  mean: -0.58008, std: 0.00315, params: {'colsample_bylevel': 0.45, 'colsample_bytree': 0.4},\n",
       "  mean: -0.58081, std: 0.00348, params: {'colsample_bylevel': 0.45, 'colsample_bytree': 0.45},\n",
       "  mean: -0.58104, std: 0.00410, params: {'colsample_bylevel': 0.45, 'colsample_bytree': 0.5},\n",
       "  mean: -0.58035, std: 0.00498, params: {'colsample_bylevel': 0.5, 'colsample_bytree': 0.3},\n",
       "  mean: -0.58072, std: 0.00406, params: {'colsample_bylevel': 0.5, 'colsample_bytree': 0.35},\n",
       "  mean: -0.58070, std: 0.00317, params: {'colsample_bylevel': 0.5, 'colsample_bytree': 0.4},\n",
       "  mean: -0.58057, std: 0.00357, params: {'colsample_bylevel': 0.5, 'colsample_bytree': 0.45},\n",
       "  mean: -0.58180, std: 0.00449, params: {'colsample_bylevel': 0.5, 'colsample_bytree': 0.5}],\n",
       " {'colsample_bylevel': 0.45, 'colsample_bytree': 0.4},\n",
       " -0.58007880617793361)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch3_1 = GridSearchCV(xgb3_1, param_grid=param_test_3_1, scoring='neg_log_loss', n_jobs=7, cv=kfold)\n",
    "gsearch3_1.fit(X_train, y_train)\n",
    "gsearch3_1.grid_scores_, gsearch3_1.best_params_, gsearch3_1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([ 190.52174058,  205.62594447,  222.88820748,  237.25411696,\n",
       "         251.22419066,  204.03436575,  223.62176914,  241.40879335,\n",
       "         259.66317863,  278.40903544,  222.75597358,  241.36470432,\n",
       "         264.22681866,  282.96629963,  306.28306527,  236.43370857,\n",
       "         259.86865244,  283.47186179,  307.26944923,  332.22155199,\n",
       "         257.82657022,  281.14255443,  309.71420465,  335.48517036,\n",
       "         333.22796803]),\n",
       " 'mean_score_time': array([ 2.60948296,  1.53091307,  1.41441221,  1.43635402,  1.54105825,\n",
       "         1.31744404,  1.46892071,  1.38276839,  1.34494119,  1.36795616,\n",
       "         1.34075618,  1.3297029 ,  1.34007797,  1.34309874,  1.35332785,\n",
       "         1.31611614,  1.3421895 ,  1.34850459,  1.38528218,  1.43055606,\n",
       "         1.4143311 ,  1.46710105,  1.45875921,  1.36967983,  1.15176992]),\n",
       " 'mean_test_score': array([-0.58086749, -0.58055257, -0.58104746, -0.58045083, -0.58028105,\n",
       "        -0.58026165, -0.58082315, -0.58050116, -0.5811154 , -0.58134522,\n",
       "        -0.58044687, -0.58075221, -0.58014409, -0.58044486, -0.58126125,\n",
       "        -0.581197  , -0.58064779, -0.58007881, -0.58081487, -0.5810397 ,\n",
       "        -0.58035071, -0.58072351, -0.58070469, -0.58057157, -0.58180217]),\n",
       " 'mean_train_score': array([-0.44981325, -0.44236326, -0.4351372 , -0.42934683, -0.42650829,\n",
       "        -0.44199592, -0.43592193, -0.42826557, -0.42410259, -0.42024717,\n",
       "        -0.43465683, -0.42930645, -0.42222601, -0.41941373, -0.4155191 ,\n",
       "        -0.43107427, -0.42499506, -0.41887734, -0.41485911, -0.41210706,\n",
       "        -0.42693112, -0.42117652, -0.41526181, -0.41201855, -0.40890969]),\n",
       " 'param_colsample_bylevel': masked_array(data = [0.3 0.3 0.3 0.3 0.3 0.35 0.35 0.35 0.35 0.35 0.4 0.4 0.4 0.4 0.4 0.45 0.45\n",
       "  0.45 0.45 0.45 0.5 0.5 0.5 0.5 0.5],\n",
       "              mask = [False False False False False False False False False False False False\n",
       "  False False False False False False False False False False False False\n",
       "  False],\n",
       "        fill_value = ?),\n",
       " 'param_colsample_bytree': masked_array(data = [0.3 0.35 0.4 0.45 0.5 0.3 0.35 0.4 0.45 0.5 0.3 0.35 0.4 0.45 0.5 0.3 0.35\n",
       "  0.4 0.45 0.5 0.3 0.35 0.4 0.45 0.5],\n",
       "              mask = [False False False False False False False False False False False False\n",
       "  False False False False False False False False False False False False\n",
       "  False],\n",
       "        fill_value = ?),\n",
       " 'params': [{'colsample_bylevel': 0.3, 'colsample_bytree': 0.3},\n",
       "  {'colsample_bylevel': 0.3, 'colsample_bytree': 0.35},\n",
       "  {'colsample_bylevel': 0.3, 'colsample_bytree': 0.4},\n",
       "  {'colsample_bylevel': 0.3, 'colsample_bytree': 0.45},\n",
       "  {'colsample_bylevel': 0.3, 'colsample_bytree': 0.5},\n",
       "  {'colsample_bylevel': 0.35, 'colsample_bytree': 0.3},\n",
       "  {'colsample_bylevel': 0.35, 'colsample_bytree': 0.35},\n",
       "  {'colsample_bylevel': 0.35, 'colsample_bytree': 0.4},\n",
       "  {'colsample_bylevel': 0.35, 'colsample_bytree': 0.45},\n",
       "  {'colsample_bylevel': 0.35, 'colsample_bytree': 0.5},\n",
       "  {'colsample_bylevel': 0.4, 'colsample_bytree': 0.3},\n",
       "  {'colsample_bylevel': 0.4, 'colsample_bytree': 0.35},\n",
       "  {'colsample_bylevel': 0.4, 'colsample_bytree': 0.4},\n",
       "  {'colsample_bylevel': 0.4, 'colsample_bytree': 0.45},\n",
       "  {'colsample_bylevel': 0.4, 'colsample_bytree': 0.5},\n",
       "  {'colsample_bylevel': 0.45, 'colsample_bytree': 0.3},\n",
       "  {'colsample_bylevel': 0.45, 'colsample_bytree': 0.35},\n",
       "  {'colsample_bylevel': 0.45, 'colsample_bytree': 0.4},\n",
       "  {'colsample_bylevel': 0.45, 'colsample_bytree': 0.45},\n",
       "  {'colsample_bylevel': 0.45, 'colsample_bytree': 0.5},\n",
       "  {'colsample_bylevel': 0.5, 'colsample_bytree': 0.3},\n",
       "  {'colsample_bylevel': 0.5, 'colsample_bytree': 0.35},\n",
       "  {'colsample_bylevel': 0.5, 'colsample_bytree': 0.4},\n",
       "  {'colsample_bylevel': 0.5, 'colsample_bytree': 0.45},\n",
       "  {'colsample_bylevel': 0.5, 'colsample_bytree': 0.5}],\n",
       " 'rank_test_score': array([18, 10, 20,  8,  4,  3, 17,  9, 21, 24,  7, 15,  2,  6, 23, 22, 12,\n",
       "         1, 16, 19,  5, 14, 13, 11, 25], dtype=int32),\n",
       " 'split0_test_score': array([-0.57444416, -0.57320461, -0.57583091, -0.57225872, -0.57409242,\n",
       "        -0.5741015 , -0.57323315, -0.57563189, -0.57516715, -0.57535659,\n",
       "        -0.57338985, -0.5747711 , -0.57486381, -0.57400028, -0.57495314,\n",
       "        -0.57372717, -0.57454381, -0.57538358, -0.57526568, -0.57429393,\n",
       "        -0.57194205, -0.5741324 , -0.57498656, -0.57485791, -0.57443659]),\n",
       " 'split0_train_score': array([-0.450386  , -0.44347761, -0.4366415 , -0.43105031, -0.42733646,\n",
       "        -0.44353142, -0.43634763, -0.4302265 , -0.42652238, -0.42179019,\n",
       "        -0.43564691, -0.42882451, -0.42457925, -0.4219994 , -0.41611522,\n",
       "        -0.43384309, -0.42447605, -0.41853778, -0.41911834, -0.41261722,\n",
       "        -0.42814443, -0.42202131, -0.4171603 , -0.41318598, -0.40950536]),\n",
       " 'split1_test_score': array([-0.58000528, -0.57938584, -0.57961718, -0.57931975, -0.5789265 ,\n",
       "        -0.5804113 , -0.58122538, -0.58015741, -0.57928258, -0.57958065,\n",
       "        -0.58035324, -0.57956768, -0.57846907, -0.57945711, -0.58052185,\n",
       "        -0.58062424, -0.57963067, -0.5786379 , -0.58057409, -0.58002092,\n",
       "        -0.57869004, -0.57969152, -0.58041269, -0.57844256, -0.58103229]),\n",
       " 'split1_train_score': array([-0.44988528, -0.44258438, -0.43572286, -0.4296793 , -0.42743043,\n",
       "        -0.44133697, -0.43544646, -0.42882168, -0.42450701, -0.42141533,\n",
       "        -0.43442915, -0.42890642, -0.42318013, -0.42102337, -0.41619125,\n",
       "        -0.43066932, -0.42534827, -0.41991976, -0.41527928, -0.41373725,\n",
       "        -0.42661825, -0.42366115, -0.4161882 , -0.41408037, -0.40934638]),\n",
       " 'split2_test_score': array([-0.58029154, -0.58070801, -0.58076615, -0.58039397, -0.57967801,\n",
       "        -0.57929191, -0.58064828, -0.57888228, -0.58021406, -0.58050641,\n",
       "        -0.57892957, -0.58094701, -0.57992715, -0.57964397, -0.58111712,\n",
       "        -0.58149574, -0.57973106, -0.57966278, -0.57961273, -0.58039768,\n",
       "        -0.5805001 , -0.57975169, -0.58148154, -0.58116909, -0.58058662]),\n",
       " 'split2_train_score': array([-0.4480143 , -0.44130746, -0.43430924, -0.42887098, -0.42633858,\n",
       "        -0.4414115 , -0.43533949, -0.42789002, -0.42157768, -0.41797649,\n",
       "        -0.43493626, -0.42927837, -0.42054852, -0.41865072, -0.41472951,\n",
       "        -0.42920759, -0.42558178, -0.4171246 , -0.412591  , -0.41066913,\n",
       "        -0.4250017 , -0.42027655, -0.41439213, -0.40971218, -0.40913632]),\n",
       " 'split3_test_score': array([-0.58383369, -0.58312331, -0.58417414, -0.58373394, -0.58343627,\n",
       "        -0.58280861, -0.58231895, -0.58361663, -0.58522664, -0.58495228,\n",
       "        -0.58357047, -0.58305913, -0.58281474, -0.58419896, -0.58370992,\n",
       "        -0.58473905, -0.58377001, -0.58197855, -0.58292488, -0.58435615,\n",
       "        -0.58444819, -0.58469144, -0.58207661, -0.58401095, -0.58579956]),\n",
       " 'split3_train_score': array([-0.45017643, -0.44239153, -0.43537716, -0.42901601, -0.42625736,\n",
       "        -0.44197186, -0.43699558, -0.42644941, -0.42351075, -0.42015446,\n",
       "        -0.43388386, -0.42973761, -0.4210824 , -0.41701634, -0.41491938,\n",
       "        -0.43099134, -0.42470559, -0.42007692, -0.41422775, -0.41253579,\n",
       "        -0.42714659, -0.41912911, -0.41430023, -0.41170808, -0.40829466]),\n",
       " 'split4_test_score': array([-0.58576427, -0.58634283, -0.58485006, -0.5865496 , -0.58527359,\n",
       "        -0.58469626, -0.58669178, -0.58421874, -0.58568795, -0.58633169,\n",
       "        -0.58599293, -0.58541752, -0.58464703, -0.58492534, -0.58600564,\n",
       "        -0.58540006, -0.58556489, -0.58473264, -0.58569847, -0.58613136,\n",
       "        -0.58617497, -0.58535191, -0.58456722, -0.58437849, -0.58715742]),\n",
       " 'split4_train_score': array([-0.45060422, -0.44205533, -0.43363524, -0.42811753, -0.42517861,\n",
       "        -0.44172785, -0.43548049, -0.42794023, -0.42439514, -0.41989939,\n",
       "        -0.43438795, -0.42978536, -0.42173976, -0.41837879, -0.41564014,\n",
       "        -0.43065999, -0.4248636 , -0.41872765, -0.4130792 , -0.41097591,\n",
       "        -0.42774463, -0.42079448, -0.41426818, -0.41140615, -0.40826571]),\n",
       " 'std_fit_time': array([  0.12274535,   1.97047935,   0.36894279,   0.09248954,\n",
       "          0.32430848,   0.87993249,   0.37193643,   0.195293  ,\n",
       "          0.38275259,   0.54671643,   0.30423194,   0.1668001 ,\n",
       "          0.37676729,   0.41744103,   0.25633079,   0.44429711,\n",
       "          0.47695425,   0.44183049,   0.42383497,   0.70232307,\n",
       "          0.30584076,   0.27968956,   1.15232418,   4.01772224,  10.9482341 ]),\n",
       " 'std_score_time': array([ 0.35956222,  0.18722989,  0.07879868,  0.09187072,  0.22607162,\n",
       "         0.03515787,  0.22345332,  0.07190721,  0.01099047,  0.07046877,\n",
       "         0.06458791,  0.00734279,  0.01650415,  0.02067911,  0.00910301,\n",
       "         0.01109483,  0.01071495,  0.00655605,  0.03260145,  0.05715331,\n",
       "         0.08909673,  0.09825255,  0.11254383,  0.02376392,  0.053311  ]),\n",
       " 'std_test_score': array([ 0.00387578,  0.00437057,  0.00327329,  0.00482416,  0.00388267,\n",
       "         0.00360724,  0.00434681,  0.00316146,  0.0039335 ,  0.00393837,\n",
       "         0.00430406,  0.00358606,  0.00341   ,  0.00393233,  0.00371113,\n",
       "         0.00415758,  0.00382248,  0.00314827,  0.00348123,  0.00409577,\n",
       "         0.00498386,  0.0040649 ,  0.00316814,  0.00357371,  0.00449394]),\n",
       " 'std_train_score': array([ 0.00093027,  0.00070725,  0.00105876,  0.00098584,  0.00082414,\n",
       "         0.00080074,  0.00064717,  0.00124095,  0.00160187,  0.00134395,\n",
       "         0.00059665,  0.00040208,  0.00147013,  0.00182633,  0.00060082,\n",
       "         0.00151614,  0.00040974,  0.00107102,  0.00232502,  0.00113567,\n",
       "         0.00109553,  0.00155144,  0.00119422,  0.00151028,  0.00052723])}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch3_1.cv_results_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "colsample_bylevel 最优值 0.45, colsample_bytree 最优值 0.4 . 当前最高得分: -0.58007880617793361 ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step IV: 再次调整弱学习器数目: n_estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "xgb4 = XGBClassifier(n_jobs=8, \n",
    "                     learning_rate=0.1, \n",
    "                     n_estimators=522, # 第 1 步得到, 进行第二轮调优\n",
    "                     max_depth=6, # 第 2 步得到\n",
    "                     min_child_weight=3, # 第 2 步得到\n",
    "                     gamma=0, \n",
    "                     subsample=1, \n",
    "                     colsample_bytree=0.4, # 第 3 步得到\n",
    "                     colsample_bylevel=0.45, # 第 3 步得到\n",
    "                     objective='multi:softprob', \n",
    "                     seed=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_fit(xgb4, X_train, y_train, cv_folds=kfold, early_stopping_rounds=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "372"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb4.get_params()['n_estimators']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEXCAYAAABCjVgAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XeYXGX1wPHvmdnek+ymbhohUoMB\nEopIF0jgJ0V6UyyAhaICElQQERQbCkoRNERFQAQVkAChJIBIS0iABEISIGXT2/Y+c35/vHc2k8ns\n7mSzM3d253ye5z5z+z1zd3bO3Pe9931FVTHGGGMAAn4HYIwxJn1YUjDGGNPBkoIxxpgOlhSMMcZ0\nsKRgjDGmgyUFY4wxHSwpGGN2iogcLiIf+h2HSQ5LCn2QiCwXkc8l+Rg3isgDyTxGMonIYyJyb8y8\nf4vI76Omh4nIfSKyRkTqReRjEZkhInt6y8eIiHrL6kVkvYjcJSLZSY5dRWT3ZB5jZ8TGo6qvqOoe\nSTrWDBG5ORn7NomxpGD6q28Bp4vI0QAicjawPzDNmx4E/A8oAA4HioEDgJeA42L2VaaqRcAE4FBv\n3yYNiUjQ7xj6OksK/YyIXCwiy0Rki4g8ISLDo5YdLyIfikiN94v3JRH5Wg+OsZeIzBGRahFZJCIn\nRy07UUTeF5E6EVktIld788tF5D/eNltE5BUR2eHzJyL3iMivYuY9LiLf9cav9fZb572XY+PFqKrr\ngKuA+0RkFHAHcKmq1nurfAeoBS5U1Y/UqVbV+1X1d53scwPwHLB3gueiVET+IiIbRWSFiPww8p5F\nZHfv/NeIyCYR+bs3/2Vv83e8q5Oz45yji0TkvyLyKxHZKiKfiMjUeDHHbFcqIn8SkbXeObw58iW6\nM/GIyFEiUhW13+Uico2IvCsiDd4xhojI097f6XkRGRC1/j9EZJ13rJdFZB9v/iXA+cD3vGM9mcA5\nniEid4vITBFpAI7u7DNoEqSqNvSxAVgOfC7O/GOATbhfvLnA74CXvWXluC/BLwBZwJVAG/C1To5x\nI/BAnPnZwDLg+0COd8w6YA9v+VrgcG98AHCAN/4z4B5v+2zcr3OJs/8jgFWRZd4+moDhwB7esuHe\nsjHAuG7O1bPeOflzzPzXgRu72XYMoECWNz0ceAf4SoLn4i/A47irkDHAEuCr3rKHgB/gfpjlAZ+N\nOq4Cu3cR10Xe3+5iIAh8A1gT73zGbPdv4A9AITAYeBOXKHcqHuAooCrm8/g6MAQYAWwA3sZdmeUC\nLwI/ilr/K945yQV+CyyIWjYDuHknPm8zgBrgsKjY434GbUhssCuF/uV8YLqqvq2qLcB1wKEiMgY4\nEVikqv9U1XbcL+d1PTjGIUARcKuqtqrqi8B/gHO95W3A3iJSoqpbVfXtqPnDgNGq2qauXDpew1uv\n4L6EDvemzwBeU9U1QAj3RbK3iGSr6nJV/aibeF8BBgGx9SPlRL1/ETnZ+yVaJyKzYtbdJCLVwGqg\nAXi0u3Ph/QI/G7hOVetUdTnwa+DCqPMxGpfgmlX1v928j1grVPU+VQ0Bf8ad2yGdrSwiQ4CpwLdV\ntUHdVc9vgHN6KZ7fqep6VV2NO+dvqOp873P4L1yCAEBVp3vnpAX34+PTIlLayX67+7wBPK6qr6pq\nWFWb6fwzaBJgSaF/GQ6siEyoKyrZjPv1Nhz3KzuyTIGq2B0keIxVqhqOmrfCOwbA6bgEtMIrjjjU\nm/9L3C++WeIqdKfF27kX18Ns+6c/D/ibt2wZ8G3cF8kGEXlYoorHYonIeOBq4C7g17J9BfFm3Bdp\n5LhPqGoZrlgpJ2ZX5d6yAuBV4JkEzkW5t58VcZYBfA8Q4E2vSOQrnb2PTnQkNFVt9EaLulh/NO5X\n91ov+VXjrhoG91I866PGm+JMF4Er8xeRW0XkIxGpxV1lgDtf8XT3eYOoz7Wns8+gSYAlhf5lDe6f\nHwARKcT9Sl6Nu6SujFom0dM7eYyRMfUBo7xjoKpvqeopuC+bfwOPePPrVPUqVd0N+Dzw3c7qA3BF\nGWeIyGjgYOCxyAJVfVBVP+u9TwV+Hm8H3vv7I6544nLcL/xro1Z5AThV4tRrdEZVm3DFFYeKSHk3\n52IT2359xy5DVdep6sWqOhy4FLhLknvH0SqgBS/BeUOJqu6T4njOA04BPgeU4orVwCUkcH/TaF1+\n3uJt09ln0CTGkkLflS0ieVFDFvAg8GURmSgiucBPcZfxy4GngAkicqq37reAod0cIxBzjFzgDdwX\n7PdEJFtEjsJ9yT8sIjkicr6IlKpqG64OIwQgIv/nVWZK1PxQvIOq6nxgI+5L/VlVrfb2sYeIHOPF\n0Yz7BRp3H7hy9nLgp96vzK96Me/pLb8NV978VxEZJ04xMLGzk+Ed90Lcr/TNXZ0Lr1jnEeAWESn2\nEtx38YqxRORMEYkk5a24L7bIe1kP7NZZHD2hqmuBWbgrphIRCXjv+8gUx1OMS06bcVdeP41ZHnus\nTs9xvJ139Rk0CfK7UsOGnR9wl9waM9zsLfs68BGwBVf2Whm13RRcZWcNrkjlNdzdN/GOcWOcY1R5\ny/bB3bpZA7wPnObNz8EVrWzF/TO+hVdhiSuWWY77B68Cru/mPV7vHfPMqHn74SpH66Le3/A4244E\nqoFDYub/CFfeHanEHg78CXcVVe+dtz8De3nLx3gx1HtDtfe+J0ftM+658JYNwCWBjbhf6jcAAW/Z\nL3C/diPHvSRqu697MVUDZ8V5fxcB/42Z12XltLdOKXC3d/5rgPnAOTsbD/Ermj8XNf0AUZX4wNeA\n573xIlzlex2uGOiL0bED44EF3rH+ncA5nsH2FdOdfgZtSGyI/HOYDONdjlcB56vqbL/jMcakBys+\nyiAicoKIlHnFIN/HleO+7nNYxpg0YkkhsxyKKxrYhCuXPVVd5anpJ2Rbkxyxw+Hdb20MVnxkjDFm\nG7tSMMYY0yHL7wB2Vnl5uY4ZM8bvMIwxpk+ZN2/eJlWt6G69PpcUxowZw9y5c/0Owxhj+hQRWdH9\nWlZ8ZIwxJoolBWOMMR0sKRhjjOnQ5+oUjDGmJ9ra2qiqqqK5udnvUJIqLy+PyspKsrN71musJQVj\nTEaoqqqiuLiYMWPG4Npl7H9Ulc2bN1NVVcXYsWN7tA8rPjLGZITm5mYGDRrUbxMCgIgwaNCgXboa\nsqRgjMkY/TkhROzqe8yYpPDxqtXMmvMyGg53v7IxxgBn/+E1zv7Da36HkVIZkxQ2zrmH4+d8nobG\nBr9DMcZkoOrqau66664ebfvb3/6WxsbG7lfsBRmTFCS3BIDG2q0+R2KMyUR9JSlkzN1HWQVeUqjb\niuvi1RhjUmfatGl89NFHTJw4keOOO47BgwfzyCOP0NLSwmmnncaPf/xjGhoaOOuss6iqqiIUCnH9\n9dezfv161qxZw9FHH015eTmzZye3T6yMSwrNDdU+R2KM8duPn1zE+2tqu13v/bVunUTqFfYeXsKP\nPr9Pp8tvvfVWFi5cyIIFC5g1axaPPvoob775JqrKySefzMsvv8zGjRsZPnw4Tz31FAA1NTWUlpZy\n2223MXv2bMrLyxN8hz2XMcVHuYVlALTW1/gciTEm082aNYtZs2ax//77c8ABB7B48WKWLl3KhAkT\neP7557n22mt55ZVXKC0tTXlsGXOlkFs4AIDWRksKxmS6rn7RR4tcIfz90kN79fiqynXXXcell166\nw7J58+Yxc+ZMrrvuOo4//nhuuOGGXj12dzLmSiGv2F0phJqs+MgYk3rFxcXU1dUBcMIJJzB9+nTq\n6+sBWL16NRs2bGDNmjUUFBRwwQUXcPXVV/P222/vsG2yZcyVQmFHUui+HNEYY3rboEGDOOyww9h3\n332ZOnUq5513Hoce6q5AioqKeOCBB1i2bBnXXHMNgUCA7Oxs7r77bgAuueQSpk6dyrBhw6yiubcU\nFLmkoM2pybbGGBPrwQcf3G76yiuv3G563LhxnHDCCTtsd/nll3P55ZcnNbaIjEkKwZw8WjQbWiwp\nGGMS09t1CX1BxtQpADRIAYE2SwrGGNOZjEoKTYECslotKRhjTGeSlhREZLqIbBCRhZ0sFxG5Q0SW\nici7InJAsmKJaA4UktVubR8ZY0xnknmlMAOY0sXyqcB4b7gEuDuJsQDQEiwkJ1Sf7MMYY0yflbSk\noKovA1u6WOUU4C/qvA6UiciwZMUD0J5VSG4oNY1KGWP6gftPckMG8bNOYQSwKmq6ypu3AxG5RETm\nisjcjRs39viAqxqzyQ1Z8ZExJvV62krqiSeeSHV16h669TMpxOseSOOtqKr3quokVZ1UUVHR4wMO\nDtRSjCUFY0zqdZYUQqFQl9vNnDmTsrKyZIW1Az+TQhUwMmq6EliTzAPqsE9TSDOhkPW+ZoxJreim\nsydPnszRRx/Neeedx4QJEwA49dRTOfDAA9lnn3249957O7YbM2YMmzZtYvny5ey1115cfPHF7LPP\nPhx//PE0NTX1epx+Prz2BHCZiDwMHAzUqOrapB4xt5hsCVFTX09paUlSD2WMSWNPT4N173W/3rp3\n3Wsi9QpDJ8DUWztdHN109pw5czjppJNYuHAhY8eOBWD69OkMHDiQpqYmJk+ezOmnn86gQYO228fS\npUt56KGHuO+++zjrrLN47LHHuOCCC7qPbSckLSmIyEPAUUC5iFQBPwKyAVT1HmAmcCKwDGgEvpys\nWCKCea4Z2oa6rZYUjDG+OuiggzoSAsAdd9zBv/71LwBWrVrF0qVLd0gKY8eOZeLEiQAceOCBLF++\nvNfjSlpSUNVzu1muwLeSdfx4gl5HO011W4HRqTy0MSaddPGLfjuRK4QvP9XrIRQWFnaMz5kzh+ef\nf57XXnuNgoICjjrqKJqbm3fYJjc3t2M8GAwmpfgoo55ozi5wVwrW+5oxJtW6av66pqaGAQMGUFBQ\nwOLFi3n99ddTHN02GdMgHkCOlxRaLSkYY1Isuuns/Px8hgwZ0rFsypQp3HPPPey3337sscceHHLI\nIb7FmVFJIddrPru1wXpfM8akXmzT2RG5ubk8/fTTcZdF6g3Ky8tZuHBbq0FXX311r8cHGZYUCopd\nl5zt1tGOMSYRSahLSHcZVaeQXxzpaMeSgjHGxJNRSWFb72uWFIzJRO6mx/5tV99jRiUFyc6jhWzE\nel8zJuPk5eWxefPmfp0YVJXNmzeTl5fX431kVJ0CQCP5iPW+ZkzGqayspKqqil1pVLMvyMvLo7Ky\nssfbZ1xSqNV8arZ21aK3MaY/ys7O3u4JYhNfRhUfAYQki6Fs8jsMY4xJSxmXFFryh5IX7L9lisYY\nsysyLim05pRSGLI6BWOMiSfjkkIot5RirevXdyAYY0xPZVxSIH8AJTTQ2NLudyTGGJN2Mi4pSP5A\nsiRMTc1Wv0Mxxpi0k3FJIavItX9UX73B50iMMSb9ZFxSyCkqB6Cp2m5LNcaYWBmXFPJKXfd2TXWb\nfY7EGGPST8YlhcLSCgDa6i0pGGNMrIxLCkVlrvgo1GBNXRhjTKyMSwp5xa74KNxoScEYY2JlXFIg\nO59G8ghYUjDGmB1kXlIA6gKlZLVYUjDGmFhJTQoiMkVEPhSRZSIyLc7y0SLygoi8KyJzRKTnjYDv\nhIbsAeS3WlIwxphYSUsKIhIE7gSmAnsD54rI3jGr/Qr4i6ruB9wE/CxZ8URryRlAYXt1Kg5ljDF9\nSjKvFA4Clqnqx6raCjwMnBKzzt7AC9747DjLk6I9bxBlWkM4bI3iGWNMtGQmhRHAqqjpKm9etHeA\n073x04BiERmUxJgA0IJyBlBLbVNrsg9ljDF9SjKTgsSZF/vT/GrgSBGZDxwJrAZ2aL5URC4Rkbki\nMrc3+lcNFFWQK+1s2WIPsBljTLRkJoUqYGTUdCWwJnoFVV2jql9Q1f2BH3jzamJ3pKr3quokVZ1U\nUVGxy4HllA4GoHbz2l3elzHG9CfJTApvAeNFZKyI5ADnAE9EryAi5SISieE6YHoS4+mQXzYEgOmz\n3krF4Ywxps9IWlJQ1XbgMuBZ4APgEVVdJCI3icjJ3mpHAR+KyBJgCHBLsuKJVjRwGACn7ZGbisMZ\nY0yfkZXMnavqTGBmzLwbosYfBR5NZgzxFA8cCkB7rfWpYIwx0TLyieasYlenQKP1qWCMMdEyMimQ\nnUcD+QQtKRhjzHYyMykAdcEysq39I2OM2U7GJoXG7AHkt231OwxjjEkrGZsUWnMHUhSy9o+MMSZa\nxiaFUN4gBmgNre1hv0Mxxpi0kbFJQQvLGUgdm+ub/Q7FGGPSRsYmhaziwWRLiC2bd70tJWOM6S8y\nNikUfPIcAFs3VPkciTHGpI+MTQrFJ3wfgKbNlhSMMSYiY5NCyZAxALRVr/Y3EGOMSSMZmxSCpcPd\nSO2arlc0xpgMkrFJgex8aqWEnAbrU8EYYyIyNykANdkVFLZYS6nGGBOR0UmhMW8Ipe12S6oxxkRk\ndFIIFQ1jsG6mpqnN71CMMSYtZHRSCG79iHKppWqDtZZqjDGQ4UmhcPIFAGxau8LnSIwxJj1kdFIo\nGzoagNr1lhSMMQYyPCkUlo8CoHmLPdVsjDGQ4UlBSkcAEK6xp5qNMQYyPCmQW0wd+bRsWeV3JMYY\nkxYyOykAzXlDGSpbUFW/QzHGGN9lfFJobWtjiG5iY32L36EYY4zvkpoURGSKiHwoIstEZFqc5aNE\nZLaIzBeRd0XkxGTGE4+OPYLRsp5VWxpTfWhjjEk7SUsKIhIE7gSmAnsD54rI3jGr/RB4RFX3B84B\n7kpWPJ3JHTyeMmlg3TprGM8YY7pNCiIyTkRyvfGjROQKESlLYN8HActU9WNVbQUeBk6JWUeBEm+8\nFEh5O9YlI/YAoGHNh6k+tDHGpJ1ErhQeA0IisjvwJ2As8GAC240Aom/rqfLmRbsRuEBEqoCZwOXx\ndiQil4jIXBGZu3Fj7zZglzt4PAAL33u7V/drjDF9USJJIayq7cBpwG9V9TvAsAS2kzjzYm/xOReY\noaqVwInAX0Vkh5hU9V5VnaSqkyoqKhI49E4YMIYwAfbO29y7+zXGmD4okaTQJiLnAl8C/uPNy05g\nuypgZNR0JTsWD30VeARAVV8D8oDyBPbde7JyqMkZSlHDCsJhuy3VGJPZEkkKXwYOBW5R1U9EZCzw\nQALbvQWMF5GxIpKDq0h+ImadlcCxACKyFy4ppLyDg+bi0YzUtayubkr1oY0xJq10mxRU9X1VvUJV\nHxKRAUCxqt6awHbtwGXAs8AHuLuMFonITSJysrfaVcDFIvIO8BBwkfrwFFmwYhxjZR1L19em+tDG\nGJNWsrpbQUTmACd76y4ANorIS6r63e62VdWZuArk6Hk3RI2/Dxy2kzH3uuLhe5K/uJEVq6pgr6F+\nh2OMMb5JpPioVFVrgS8A96vqgcDnkhtWauUPcXcg1a5e7HMkxhjjr0SSQpaIDAPOYltFc/8yaBwA\noc3LfA7EGGP8lUhSuAlXL/CRqr4lIrsBS5MbVooNGENIsiiuXUbI7kAyxmSwRCqa/6Gq+6nqN7zp\nj1X19OSHlkLBbGqLxzFeV1obSMaYjJZIMxeVIvIvEdkgIutF5DERqUxFcKmkQ/Zlz8BKFq2xO5CM\nMZkrkeKj+3HPFwzHNVPxpDevXykZPZGhspXFH33idyjGGOObRJJCharer6rt3jAD6OW2JvyXNWwC\nALUrF/gciTHG+CeRpLBJRC4QkaA3XAD0v4aChuwLQO6m92ltD/scjDHG+CORpPAV3O2o64C1wBm4\npi/6l6IKNssAxusKPlhr9QrGmMyUyN1HK1X1ZFWtUNXBqnoq7kG2fqdo1KfZK7CS+Su3+h2KMcb4\noqc9r3XbxEVflDtiPz4VqOKdlZv8DsUYY3zR06QQr6+Evm/Js+TQzopFb/odiTHG+KKnSaF/PvZ7\n4T8B2E8XU7XVHmIzxmSeTpOCiNSJSG2coQ73zEL/U1pJW9EIJgWW8PrHW/yOxhhjUq7TpKCqxapa\nEmcoVtVum9zuq7LCrUwOLObWme/7HYoxxqRcT4uP+i056lqGSDXFzWvxob8fY4zxlSWFWCMPBly9\nwqot1j2nMSazWFKINWQfQtlFTAos4bWP7dZUY0xmsaQQKxAkEMzikMAH3Pq09cRmjMksiTSdHe8u\npFVec9q7pSLIVJNDvsm4wBpy2utobgv5HY4xxqRMIlcKtwHX4JrNrgSuBu4DHgamJy80H406mADK\nnu0fcvLv/+t3NMYYkzKJJIUpqvoHVa1T1VpVvRc4UVX/DgxIcnz+GDEJBQ4LLGRrQ6vf0RhjTMok\nkhTCInKWiAS84ayoZf3zns3cImTUZzix8ENCCu0ha0rbGJMZEkkK5wMXAhu84ULgAhHJBy7rakMR\nmSIiH4rIMhGZFmf5b0RkgTcsEZHqHryH5PjU8VS2LCO7YR2vftT/uo8wxph4Emk6+2NV/byqlnvD\n51V1mao2qWqnBe4iEgTuBKYCewPnisjeMfv+jqpOVNWJwO+Af+7a2+lF448H4KSst3jojZU+B2OM\nMamRyN1Hld6dRhtEZL2IPCYilQns+yBgmZdUWnEV06d0sf65wEOJhZ0Cg/eGkkqOzPmAZxetY0Nt\ns98RGWNM0iVSfHQ/8ASuEbwRwJPevO6MAFZFTVd583YgIqOBscCLnSy/RETmisjcjRs3JnDoXiAC\nnzqewwMLyaaNf8yrSs1xjTHGR4kkhQpVvV9V271hBlCRwHbx+lzorGL6HOBRVY37UICq3quqk1R1\nUkVFIofuJStfJ9DeyAl5i7j9+aXWd7Mxpt9LJClsEpELRCToDRcAidS8VgEjo6YrgTWdrHsO6VR0\nFHHpy1AwiOtHLaQ1FObf81f7HZExxiRVIknhK8BZwDpgLXAG8OUEtnsLGC8iY0UkB/fF/0TsSiKy\nB+55h9cSDTplgtkQzKFi5dMMyWnhhscXEgr3z7twjTEGErv7aKWqnqyqFao6WFVPBb6QwHbtuFtW\nnwU+AB5R1UUicpOInBy16rnAw5qu7VSf/TcE5Y+TV9PcHub437zkd0TGGJM0Pe0s57vAb7tbSVVn\nAjNj5t0QM31jD2NIjREHwKDd2XfjU+w17AfUt7TR0h4iNyvod2TGGNPretpKarxK5P5JBA74IrLy\nNX5yqLBqSxPH/GqO31EZY0xS9DQppGdRT7LsfyFk5TFp/T8YVJjDmupmFq6u8TsqY4zpdZ0mhU6a\nzK4VkTrcMwuZo2AgTDgT3n2EF781kaygcOY9r9HSbs1qG2P6l06TgqoWq2pJnKFYVXtaF9F3HXwp\ntDVS+v7fuPeLk2hqC/Gb55b6HZUxxvQq63ktUUMnwG5Hw2t3cvRuxVQU5XLPSx8xb8UWvyMzxphe\nY0lhZxz5PWjYAHcezItXH0luVoBz7n2dtTVNfkdmjDG9wpLCzhj9GdjjJGjcTHHbZp647LOEwsqx\nv36JhpZ2v6MzxphdZklhZx3/E2hvgRd+wh5Dixk/uIjG1hBXPDTfOuMxxvR5lhR21qBxUDQYFjwA\na+bz7HeOZMygAl5YvIGDf/qCNYNhjOnTLCn0xDdfh8IKeHoaqDLnmqMZOSCfzQ2tTL7leZpa7VZV\nY0zfZEmhJ/JK4JjrYdXr8N6jALxy7TH84MS92NLQyoE3P8eqLY0+B2mMMTvPkkJP7X8B5BTBv78O\ndesBuPiI3Zh+0SQaW0Mc/ovZPP/+ep+DNMaYnWNJoacCQbj4RQiH4K6DwWvk9Zg9h/DSNUdRkBPk\na3+Zy4+fXERjq92ZZIzpGywp7IqKPWDAGGjaCvP/2jF79KBC3r7+OIaU5HL/q8vZ/6bneHlJiroR\nNcaYXWBJYVdd/jbklsKTV8K6hR2z87KDvPH9z/HIpYfSFgrzxelvcuXD862uwRiT1iwp7KpAAL71\nBgSy4L5joG7ddosPGjuQD34yhRFleTy+YA1H/GI2Nzy+kPW1zT4FbIwxnZN07fCsM5MmTdK5c+f6\nHcaO1r4D9x4F2QVw9RLIKdxxlZom7nhhGQ+/uRIFBhfn8uDFB7P74OKUh2uMySwiMk9VJ3W7niWF\nXvTh0/DQOZA3wCWGrJy4q63Y3MAZd7/GxvoWAIrzsrj51H2Zsu9Q69HNGJMUlhT8Mu/P8OQVUDAI\nrloCwc5bGd9U38I/5lbx2+eX0NIeRoCvHT6Wcw4axbiKotTFbIzp9ywp+On2ibD1E9jvbDjlri4T\nA0A4rLz60SaueGg+WxvbAHf1MG3qnhyz52CGleanImpjTD9mScFvL/8SXrwZPjUVzpgOOQUJbbah\nrplH51Vx+/NLaWl3DewV5AT54qFj+My4QUwaM4CCnMzr48gYs2ssKaSDN++DmVdDbjFc+a7r1jNB\nqsqyDfW8uHgDv5+9jLrm7R+AG1qSy/em7MmEEaXsVlFEMCC9Hb0xph+xpJAu3n8CHvkiZOXB11+B\n8vE92k1DSztzV2xl2mPvsr62mdjGWIMC5cW5fO+EPZlQWco4SxTGmChpkRREZApwOxAE/qiqt8ZZ\n5yzgRkCBd1T1vK722eeSAsDyV+EvJ7umME77A+x35i7vMhRWPtpYz9f/Oo9VWxsJh5VQzJ8yIFBR\nnMtVx+3BHkOL2X1wEYW5VvRkTCbyPSmISBBYAhwHVAFvAeeq6vtR64wHHgGOUdWtIjJYVTd0td8+\nmRQAalbDXYdASy0UDYEr34Hs3q1ADoWVjzfW897qGn7xzGI21rXskChEICDC4OJcvnPcp6gsy2dY\nWT7DSvPIy7bbYY3pr9IhKRwK3KiqJ3jT1wGo6s+i1vkFsERV/5jofvtsUgAItcPsW+C/t7mH3L48\nE4bvn9RDhsPKyi2NLF5Xy83/+YB1tc2EVXcofooIeElDBIaW5HHNlD0ZUZbH0NJ8Kopyycmyh+CN\n6YvSISmcAUxR1a950xcCB6vqZVHr/Bt3NXEYrojpRlV9pqv99umkELH0efjbGYDCYVfC0T+ArNyU\nhtAeCrO6uonV1U2srW7mun+9R1t7mEBA0C6SRkQwIAh0JI8rjh3PwMIcBhTmMLAgh4riXCuqMiaN\npENSOBM4ISYpHKSql0et8x+gDTgLqAReAfZV1eqYfV0CXAIwatSoA1esWJGUmFOquQZ+Pxnq17ur\nhvMegbGH+x3Vdmqb21hb3cwoHdANAAATw0lEQVSamibW1zTzuxeXsaa6CcVdUSgdLYZ3SmTbOgML\nc5iy71BK8rIpyc/yXrMpycvyXrfNt6IsY3pXOiSFRIqP7gFeV9UZ3vQLwDRVfauz/faLK4VoS2bB\nzKugeiVMPB9O+Cnkl/kd1U5pbG1nc30rWxrcsLmhlU31Lcx49RM21bfS7l12iADqksnOCAiIuCuT\ngYU5nDmpkpK8bApzsyjICVKQEyQ/x43nZwfJ9+YVZGeRlxOwpkOMIT2SQhauaOhYYDWuovk8VV0U\ntc4UXOXzl0SkHJgPTFTVzZ3tt98lBYC2JnjpF/Dq7a4S+vO3w6eO9zuqpFFVWtrD1Da1UdvcRk1T\nO7VNbdQ0tVHX3EZtczsPvL6CdTXNKK6oSlUTujLpighEbtINq9vvlH2GkpsVIDc76L26JJLXxWtO\nMEBOlhtyswLkBIMd0zlZbnl2UBCxW4JN+vA9KXhBnAj8FldfMF1VbxGRm4C5qvqEuP+aXwNTgBBw\ni6o+3NU++2VSiFj9Nsw4CdoaYcJZcNyPoWS431GlHVWloTVEQ0s7ja0hmlpDNLVFj4e2G69vaaeh\npZ365nZmf7iB2uZ2Qt7VS6LFYL0hNimBe74E6LhLLFJXA3TEGAwKA/JzOH6fIWQHAwQDQlZQyAoI\nwUCA7IAQDArZgehlAW+5EAi4mweCASEgEjUOgYAQ9KZF6BgPeOsGxW0fu23QuxkhGIje77abFMSb\nlqj57nxvP98SZ+qkRVJIhn6dFADaW1wTGS//yn2LHHktHPJNyCvxO7KMEbmSaWkL09IeorktTHN7\niBbvtbU9TGt7mJb2MK2hcMd0a3tou+kWb/ypd9eypbG1o9xMifrCD7hKl0hSiH7eMJI4IrP61n9q\n8kSnEY2dF1WHFUnC4S7ObWRebJKG7RM1xPzNIut487LizEtku51ZB2D3wUU88+0j6AlLCn3d1uVw\n37HQuMn1B/3Zq+CQb+xUUxmm/1NVQmGl3RtCIaUtHCYUVtpCYcJhCHnrqGrHeGR+WJVw2N1tFgq7\n6cirG2fHbTVq+7B6+3G3P0fuXFMvtrAqqu5LN+x910SOp3h3uUW294oII9tE9vXUu2vY3NBKu/ct\nnRX1zR2ZFwxKR4aI94WrbP/F3b4zX8rerhNJLvHmdST3qHU0Zl7sdGfzdisv5IWrjqInLCn0F2vm\nw19PhyavmqV4OFz8ghUrGWN2SqJJwZ5ESnfD94drP4ZvvAb7nQN1a+C2veDxb8H6Rd1vb4wxO8Gu\nFPqarStg+glQt9ZN55XBF+6F3Y9z/UUbY0wciV4p2COnfc2A0XDVYmjcAm//GZ7/MTx4lmuF9bBv\nw4QzoXx3v6M0xvRRdqXQ14Xa4P3H4anvuqekAXKKYMqtLkFk5/kbnzEmLVhFcyaqXQML/+ka3Wtr\nhECWu3rY/wIYONbv6IwxPrKkkMlU4ZOX4Y174MOZbl5uCfzfb2Cvz6e88T1jjP+sTiGTicBuR7ph\n6wpY9E+Ycys89lV39XDAF90T0yMPtsppY8x27EohU4TD8PGLsOAhlyQ0DIhLEHueBGOPtPoHY/ox\nKz4ynWupgw+fgWemuSemIwoGwfG3wKdOsCenjelnLCmYxLS3wCevuIfh6tfT0VZAbgkcfhWMOxqG\n7Oua2jDG9FmWFMzOU3XNanw4E167093BFJE/EI64xnUENHgfq4swpo+xpGB2Xe1adxfTc9dDw0av\nHsJTUA7H3gDjjoGykf7FaIxJiCUF0/tqqlxR03M3uCQRKWqSAEy+GEZ/xg1Fg30N0xizI0sKJrlU\nYeNi+OhF12tcc1S32hKAwsEw9VYYczgUlvsXpzEGsKRgUq29Fda+Ayv/By/cBOH27ZcHsmHKz2D4\nATB0X3uAzpgUs6Rg/BVqd5XWn7wEc362Y5IAlyhO/h2MOAAGjbfKa2OSyJKCSS+qULsaVs+DZ65z\n47EkCEddB6MOgREHQk5B6uM0pp+yZi5MehGB0ko37H2KmxcOwaalLlE8eSWE22D2zVHbBKF4KEz9\nOQydAGWjt++b0BjT6+xKwaSXpq2w6k1Y8T/43+9AQzuuE8yBsx+Aysn25LUxCbLiI9M/tDXD+oWw\n7j2Y83OoX7vjOoEs9/T1sImu+9KSYamP05g0Z0nB9F+tjbB6LqxZ4N3p1BZ/vdJRcNKvXLIoHpLa\nGI1JM5YUTGZpbXBXE2vmw6zr4yeK/IFwyDfc1cSwiVBUkfo4jfFJWiQFEZkC3A4EgT+q6q0xyy8C\nfglEbkX5var+sat9WlIwCWupc4ni3990T2NraPumOgDyB8Gh33BJYsi+rmLbKrNNP+R7UhCRILAE\nOA6oAt4CzlXV96PWuQiYpKqXJbpfSwpmlzTXwrp3XauwNVUuSWyXKMQ9kV00GI7+vrvrqWIv62vC\n9HnpcEvqQcAyVf3YC+hh4BTg/S63MiaZ8kpgzGfhyne2zWuqhvWLXIX2y7+Gxo1QtxaeuHzbOhJw\nt8iWDIcTfgrln3L9XgezU/8ejEmiZCaFEcCqqOkq4OA4650uIkfgriq+o6qrYlcQkUuASwBGjRqV\nhFBNRssvgzGHueHgS928cBi2fuKKn579obvrKdwO1Svg7+dvv30gCw67EgbvDYP3gkG7WzMeps9K\nZlKIVzAbW1b1JPCQqraIyNeBPwPH7LCR6r3AveCKj3o7UGN2EAjAoHFu2OfUbfNb6mDTEti4xL3+\n7w6XLF759Y77kKArhjrsShi4mxvKRkNWTurehzE7KZlJoQqIbmi/ElgTvYKqbo6avA/4eRLjMWbX\n5Ra7JjhGHOimP/cj99reApuXwYYP3DBvBjRtccVQz0yL2Ylsa0n2sMu3JYwBY+wKw/gumRXNWbgi\noWNxdxe9BZynqoui1hmmqmu98dOAa1X1kK72axXNpk9RhcbNsOVj2PyRe517v6u3iEsgrxQmnOEa\nCRy0O5TvDqUjrUtUs0t8r2hW1XYRuQx4FndL6nRVXSQiNwFzVfUJ4AoRORloB7YAFyUrHmN8IeL6\nkygsh5EHuXnH/MC9qrpmPbZLGH9ySeStTu7MDmTBZy53ySKSNAoHpea9mIxgD68Zk45UXe92m5e5\nYdNSeP2u+E2Qg3d3VAAKK1zSKBsNA0a717yS1MZu0pLvzykkiyUFk/FC7VCzEjZ5CePVO1xx1A7P\nXMQoGAQTz/eSxRj3WjrSnsHIEJYUjMlEkSKprcvd7bNbl8OLt7hmPyTQSdIQQF2nR8f80N1xNXCc\new4jOz+18ZuksaRgjNlROOzuiKpe6ZJG9Up48z5o2ND5NhJ0dSKTL4bSEa5PjJIR7irDbq/tMywp\nGGN2XlP1torvF25yCURD7gpkh8eMohQNhc9c5m6rLR0JZaMgf4C1I5VGLCkYY3pXW7PrRrWmyg3V\nK9xDe51VfsO2CvCCCjjk617veyOhbCQUDbHbbFPIkoIxJrUat3hFUqtcsdRrv3d3UHWVNBD3QOAe\nJ27rrrVspEscpZWQU5iy8Ps7SwrGmPTSXLvtKqNmFbxyG9Svc0VT8bpdjZAA5JXBhDO9hFG5LWkU\nDnZNkphuWVIwxvQtoXZXhxFJHC/+BGrXANr97ba5pbDX/2272iitdD3vlY6wO6g8vj/RbIwxOyWY\n5a4Eyrwm0/Y7c/vlqtBcs/3VxrPfh1AbtNbDggfptDJcArDnSduuMKJfC8utQjyKJQVjTN8g4po5\nzy+Dofu6eQddvP06oTZ3dRFJGjWrYM7P3XMai5/q+moDoHAITP7q9sVUJSMy6tZbKz4yxmSOyMN9\nHUmjCl69fVsxVeRBvh1EHvDLgv0vdN22Fg12t+IWD4GSStfESBrXb1idgjHG9ETHrber3J1Utath\n7nSoX++t0Fni8JZJAAoGwgFfcnUaJZXe6wjXAq5PRVWWFIwxJllCbVC/wd09VbfOK7JaBfMf7KJZ\ndI8EAHFtUe1/gevitXgYlAyD4uHuCiQJz29YUjDGGD+F2l3SqFkNtVVe4lgN7z3imkfvUlRx1YFf\n3pYwRh3sOmTqAUsKxhiT7sJhaNzkEkbd2qjXtfDOQzs+vzFwd7hiXo8OZbekGmNMugsEvArrwcDE\n7Zedeue28dZGlyzyByQ9JEsKxhiT7nIKXJPmKZC+908ZY4xJOUsKxhhjOlhSMMYY08GSgjHGmA6W\nFIwxxnSwpGCMMaaDJQVjjDEdLCkYY4zp0OeauRCRjcCKHm5eDmzqxXCSwWLsHRZj77AYe0c6xDha\nVSu6W6nPJYVdISJzE2n7w08WY++wGHuHxdg7+kKMEVZ8ZIwxpoMlBWOMMR0yLSnc63cACbAYe4fF\n2Dssxt7RF2IEMqxOwRhjTNcy7UrBGGNMFywpGGOM6ZAxSUFEpojIhyKyTESm+R1PhIgsF5H3RGSB\niMz15g0UkedEZKn3mvzulraPabqIbBCRhVHz4sYkzh3eeX1XRA7wMcYbRWS1dy4XiMiJUcuu82L8\nUEROSFGMI0Vktoh8ICKLRORKb37anMsuYkybcykieSLypoi848X4Y2/+WBF5wzuPfxeRHG9+rje9\nzFs+xscYZ4jIJ1HncaI335f/m4Soar8fgCDwEbAbkAO8A+ztd1xebMuB8ph5vwCmeePTgJ+nOKYj\ngAOAhd3FBJwIPI3rafwQ4A0fY7wRuDrOunt7f/NcYKz3WQimIMZhwAHeeDGwxIslbc5lFzGmzbn0\nzkeRN54NvOGdn0eAc7z59wDf8Ma/CdzjjZ8D/D0F57GzGGcAZ8RZ35f/m0SGTLlSOAhYpqofq2or\n8DBwis8xdeUU4M/e+J+BU1N5cFV9GdiSYEynAH9R53WgTESG+RRjZ04BHlbVFlX9BFiG+0wklaqu\nVdW3vfE64ANgBGl0LruIsTMpP5fe+aj3JrO9QYFjgEe9+bHnMXJ+HwWOFRHxKcbO+PJ/k4hMSQoj\ngFVR01V0/cFPJQVmicg8EbnEmzdEVdeC+6cFBvsW3TadxZRu5/Yy73J8elSxm+8xekUY++N+Qabl\nuYyJEdLoXIpIUEQWABuA53BXKNWq2h4njo4YveU1wKBUx6iqkfN4i3cefyMiubExxonfV5mSFOL9\nSkiXe3EPU9UDgKnAt0TkCL8D2knpdG7vBsYBE4G1wK+9+b7GKCJFwGPAt1W1tqtV48xLSZxxYkyr\nc6mqIVWdCFTirkz26iKOtIhRRPYFrgP2BCYDA4Fr/YwxEZmSFKqAkVHTlcAan2LZjqqu8V43AP/C\nfeDXRy4lvdcN/kXYobOY0ubcqup67x8zDNzHtmIN32IUkWzcl+3fVPWf3uy0OpfxYkzHc+nFVQ3M\nwZXDl4lIVpw4OmL0lpeSeFFjb8Y4xSueU1VtAe4nTc5jVzIlKbwFjPfuVsjBVT494XNMiEihiBRH\nxoHjgYW42L7krfYl4HF/ItxOZzE9AXzRu5viEKAmUjSSajFlsqfhziW4GM/x7koZC4wH3kxBPAL8\nCfhAVW+LWpQ257KzGNPpXIpIhYiUeeP5wOdwdR+zgTO81WLPY+T8ngG8qF7tbopjXByV/AVX5xF9\nHtPi/2YHftd0p2rA1fYvwZVF/sDveLyYdsPdyfEOsCgSF6788wVgqfc6MMVxPYQrMmjD/aL5amcx\n4S6D7/TO63vAJB9j/KsXw7u4f7phUev/wIvxQ2BqimL8LK5I4F1ggTecmE7nsosY0+ZcAvsB871Y\nFgI3ePN3wyWkZcA/gFxvfp43vcxbvpuPMb7onceFwANsu0PJl/+bRAZr5sIYY0yHTCk+MsYYkwBL\nCsYYYzpYUjDGGNPBkoIxxpgOlhSMMcZ0sKRgjDGmgyUFYxIgIhNjmo8+WXqpCXYR+baIFPTGvozZ\nVfacgjEJEJGLcA8YXZaEfS/39r1pJ7YJqmqot2Mxxq4UTL8iImPEdRhzn9fZySyv2YF4644TkWe8\nFmpfEZE9vflnishCr8OUl72mUW4CzvY6SjlbRC4Skd97688QkbvFdVbzsYgc6bUs+oGIzIg63t0i\nMle274TlCmA4MFtEZnvzzhXX8dJCEfl51Pb1InKTiLwBHCoit4rI+14LnL9Kzhk1GcfvR6ptsKE3\nB2AM0A5M9KYfAS7oZN0XgPHe+MG4NnLANTswwhsv814vAn4ftW3HNK4jlYdxTRecAtQCE3A/uuZF\nxRJpziKIazBtP296OV5HS7gEsRKoALJwzSSc6i1T4KzIvnDNTEh0nDbYsKuDXSmY/ugTVV3gjc/D\nJYrteE1Ffwb4h9cG/h9wvZABvArMEJGLcV/giXhSVRWXUNar6nvqWhhdFHX8s0TkbVwbOfvgejGL\nNRmYo6ob1fUF8DdcL3MAIVxrpuASTzPwRxH5AtCYYJzGdCmr+1WM6XNaosZDQLziowCuk5aJsQtU\n9esicjBwEtDRr26CxwzHHD8MZHktil4NTFbVrV6xUl6c/XTVQ1izevUIqtouIgcBx+Ja/b0M1xOZ\nMbvErhRMRlLXkcwnInImdHSk/mlvfJyqvqGqNwCbcO3e1+H6MO6pEqABqBGRIbhOlSKi9/0GcKSI\nlItIEDgXeCl2Z96VTqmqzgS+jesMx5hdZlcKJpOdD9wtIj/E9an7MK4Z81+KyHjcr/YXvHkrgWle\nUdPPdvZAqvqOiMzHFSd9jCuiirgXeFpE1qrq0SJyHa6vAAFmqmq8/jSKgcdFJM9b7zs7G5Mx8dgt\nqcYYYzpY8ZExxpgOVnxk+j0RuRM4LGb27ap6vx/xGJPOrPjIGGNMBys+MsYY08GSgjHGmA6WFIwx\nxnSwpGCMMabD/wNYVL/p2q9U2wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x112438358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 读取结果\n",
    "cv_result = pd.DataFrame.from_csv('1_nestimators.csv')\n",
    "# 绘图\n",
    "test_means = cv_result['test-mlogloss-mean']\n",
    "test_stds = cv_result['test-mlogloss-std']\n",
    "\n",
    "train_means = cv_result['train-mlogloss-mean']\n",
    "train_stds = cv_result['train-mlogloss-std']\n",
    "\n",
    "x_axis = range(0, cv_result.shape[0])\n",
    "\n",
    "plt.errorbar(x_axis, test_means, yerr=test_stds, label='test')\n",
    "plt.errorbar(x_axis, train_means, yerr=train_stds, label='train')\n",
    "plt.title('Log Loss vs XGBoost n_estimators')\n",
    "plt.xlabel('n_estimators')\n",
    "plt.ylabel('Log Loss')\n",
    "plt.legend(loc='best')\n",
    "plt.savefig('2_n_estimators.png')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.58288499999999999"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.min(test_means)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第二轮参数调整得到的 n_estimators 最优值为 372. 虽然弱学习器数目减少了, 但得分却降低了, 需要继续调优其他参数."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step V: 调整正则化参数: reg_alpha 和 reg_lambda"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "xgb5_1 = XGBClassifier(n_jobs=6, \n",
    "                     learning_rate=0.1, \n",
    "                     n_estimators=372, # 第 4 步得到\n",
    "                     max_depth=6, # 第 2 步得到\n",
    "                     min_child_weight=3, # 第 2 步得到\n",
    "                     gamma=0, \n",
    "                     subsample=1, \n",
    "                     colsample_bytree=0.4, # 第 3 步得到\n",
    "                     colsample_bylevel=0.45, # 第 3 步得到\n",
    "                     objective='multi:softprob', \n",
    "                     seed=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'reg_alpha': [0.1, 1, 1.5, 2], 'reg_lambda': [0.1, 0.5, 1, 2]}"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 参数调整范围暂依老师示例\n",
    "reg_alpha = [0.1, 1, 1.5, 2]\n",
    "reg_lambda = [0.1, 0.5, 1, 2]\n",
    "param_test5_1 = dict(reg_alpha=reg_alpha, reg_lambda=reg_lambda)\n",
    "param_test5_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.58048, std: 0.00347, params: {'reg_alpha': 0.1, 'reg_lambda': 0.1},\n",
       "  mean: -0.57964, std: 0.00341, params: {'reg_alpha': 0.1, 'reg_lambda': 0.5},\n",
       "  mean: -0.58025, std: 0.00339, params: {'reg_alpha': 0.1, 'reg_lambda': 1},\n",
       "  mean: -0.57967, std: 0.00279, params: {'reg_alpha': 0.1, 'reg_lambda': 2},\n",
       "  mean: -0.57916, std: 0.00304, params: {'reg_alpha': 1, 'reg_lambda': 0.1},\n",
       "  mean: -0.57935, std: 0.00368, params: {'reg_alpha': 1, 'reg_lambda': 0.5},\n",
       "  mean: -0.57914, std: 0.00288, params: {'reg_alpha': 1, 'reg_lambda': 1},\n",
       "  mean: -0.57926, std: 0.00344, params: {'reg_alpha': 1, 'reg_lambda': 2},\n",
       "  mean: -0.57886, std: 0.00322, params: {'reg_alpha': 1.5, 'reg_lambda': 0.1},\n",
       "  mean: -0.57912, std: 0.00350, params: {'reg_alpha': 1.5, 'reg_lambda': 0.5},\n",
       "  mean: -0.57934, std: 0.00317, params: {'reg_alpha': 1.5, 'reg_lambda': 1},\n",
       "  mean: -0.57914, std: 0.00303, params: {'reg_alpha': 1.5, 'reg_lambda': 2},\n",
       "  mean: -0.57901, std: 0.00309, params: {'reg_alpha': 2, 'reg_lambda': 0.1},\n",
       "  mean: -0.57930, std: 0.00286, params: {'reg_alpha': 2, 'reg_lambda': 0.5},\n",
       "  mean: -0.57922, std: 0.00360, params: {'reg_alpha': 2, 'reg_lambda': 1},\n",
       "  mean: -0.57965, std: 0.00326, params: {'reg_alpha': 2, 'reg_lambda': 2}],\n",
       " {'reg_alpha': 1.5, 'reg_lambda': 0.1},\n",
       " -0.57885925187445653)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch5_1 = GridSearchCV(xgb5_1, param_grid=param_test5_1, scoring='neg_log_loss', cv=kfold, n_jobs=6)\n",
    "gsearch5_1.fit(X_train, y_train)\n",
    "gsearch5_1.grid_scores_, gsearch5_1.best_params_, gsearch5_1.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "reg_alpha 最优值为 1.5, reg_lambda 最优值为 0.1 在参数范围边缘取得, 考虑进一步确定最优 reg_lambda. 最高得分: -0.57885925187445653 ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "xgb5_2 = XGBClassifier(n_jobs=6, \n",
    "                     reg_alpha=1.5, \n",
    "                     learning_rate=0.1, \n",
    "                     n_estimators=372, \n",
    "                     max_depth=6, \n",
    "                     min_child_weight=3, \n",
    "                     gamma=0, \n",
    "                     subsample=1, \n",
    "                     colsample_bytree=0.4, \n",
    "                     colsample_bylevel=0.45, \n",
    "                     objective='multi:softprob', \n",
    "                     seed=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'reg_lambda': [0.01, 0.02, 0.05, 0.1, 0.2]}"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reg_lambda = [0.01, 0.02, 0.05, 0.1, 0.2]\n",
    "param_test5_2 = dict(reg_lambda=reg_lambda)\n",
    "param_test5_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.57867, std: 0.00345, params: {'reg_lambda': 0.01},\n",
       "  mean: -0.57965, std: 0.00349, params: {'reg_lambda': 0.02},\n",
       "  mean: -0.57898, std: 0.00332, params: {'reg_lambda': 0.05},\n",
       "  mean: -0.57886, std: 0.00322, params: {'reg_lambda': 0.1},\n",
       "  mean: -0.57886, std: 0.00354, params: {'reg_lambda': 0.2}],\n",
       " {'reg_lambda': 0.01},\n",
       " -0.57867147354531878)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch5_2 = GridSearchCV(xgb5_2, param_grid=param_test5_2, scoring='neg_log_loss', cv=kfold, n_jobs=6)\n",
    "gsearch5_2.fit(X_train, y_train)\n",
    "gsearch5_2.grid_scores_, gsearch5_2.best_params_, gsearch5_2.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "reg_lambda 最优值还是在参数范围边缘取得, 继续缩小范围. 同时 reg_alpha 也继续调优."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'reg_alpha': [1.4, 1.5, 1.6], 'reg_lambda': [0.002, 0.005, 0.01]}"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reg_alpha = [1.4, 1.5, 1.6]\n",
    "reg_lambda = [0.002, 0.005, 0.01]\n",
    "param_test5_3 = dict(reg_alpha=reg_alpha, reg_lambda=reg_lambda)\n",
    "param_test5_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "xgb5_3 = XGBClassifier(n_jobs=6,  \n",
    "                     learning_rate=0.1, \n",
    "                     n_estimators=372, \n",
    "                     max_depth=6, \n",
    "                     min_child_weight=3, \n",
    "                     gamma=0, \n",
    "                     subsample=1, \n",
    "                     colsample_bytree=0.4, \n",
    "                     colsample_bylevel=0.45, \n",
    "                     objective='multi:softprob', \n",
    "                     seed=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.57906, std: 0.00348, params: {'reg_alpha': 1.4, 'reg_lambda': 0.002},\n",
       "  mean: -0.57875, std: 0.00334, params: {'reg_alpha': 1.4, 'reg_lambda': 0.005},\n",
       "  mean: -0.57907, std: 0.00328, params: {'reg_alpha': 1.4, 'reg_lambda': 0.01},\n",
       "  mean: -0.57881, std: 0.00309, params: {'reg_alpha': 1.5, 'reg_lambda': 0.002},\n",
       "  mean: -0.57905, std: 0.00319, params: {'reg_alpha': 1.5, 'reg_lambda': 0.005},\n",
       "  mean: -0.57867, std: 0.00345, params: {'reg_alpha': 1.5, 'reg_lambda': 0.01},\n",
       "  mean: -0.57898, std: 0.00312, params: {'reg_alpha': 1.6, 'reg_lambda': 0.002},\n",
       "  mean: -0.57902, std: 0.00333, params: {'reg_alpha': 1.6, 'reg_lambda': 0.005},\n",
       "  mean: -0.57892, std: 0.00326, params: {'reg_alpha': 1.6, 'reg_lambda': 0.01}],\n",
       " {'reg_alpha': 1.5, 'reg_lambda': 0.01},\n",
       " -0.57867147354531878)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch5_3 = GridSearchCV(xgb5_3, param_grid=param_test5_3, scoring='neg_log_loss', cv=kfold, n_jobs=6)\n",
    "gsearch5_3.fit(X_train, y_train)\n",
    "gsearch5_3.grid_scores_, gsearch5_3.best_params_, gsearch5_3.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "确定 reg_alpha 最优值为 1.5, reg_lambda 最优值为 0.01, 最高得分: -0.57867147354531878 ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step VI: 尝试调整 gamma 参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "gamma = [0, 0.001, 0.002, 0.005, 0.01]\n",
    "param_test6 = dict(gamma=gamma)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb6 = XGBClassifier(n_jobs=6,  \n",
    "                     learning_rate=0.1, \n",
    "                     n_estimators=372, # 第 4 步得到\n",
    "                     max_depth=6, # 第 2 步得到\n",
    "                     min_child_weight=3, # 第 2 步得到\n",
    "                     gamma=0, \n",
    "                     subsample=1, \n",
    "                     colsample_bytree=0.4, # 第 3 步得到\n",
    "                     colsample_bylevel=0.45, # 第 3 步得到\n",
    "                     reg_alpha=1.5, # 第 5 步得到\n",
    "                     reg_lambda=0.01, # 第 5 步得到\n",
    "                     objective='multi:softprob', \n",
    "                     seed=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/qi/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.57867, std: 0.00345, params: {'gamma': 0},\n",
       "  mean: -0.57896, std: 0.00312, params: {'gamma': 0.001},\n",
       "  mean: -0.57899, std: 0.00304, params: {'gamma': 0.002},\n",
       "  mean: -0.57890, std: 0.00320, params: {'gamma': 0.005},\n",
       "  mean: -0.57877, std: 0.00308, params: {'gamma': 0.01}],\n",
       " {'gamma': 0},\n",
       " -0.57867147354531878)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch6 = GridSearchCV(xgb6, param_grid=param_test6, scoring='neg_log_loss', cv=kfold, n_jobs=6)\n",
    "gsearch6.fit(X_train, y_train)\n",
    "gsearch6.grid_scores_, gsearch6.best_params_, gsearch6.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "结论: gamma 不建议调整, 尝试值得分都下降."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 调用模型进行测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'base_score': 0.5,\n",
       " 'booster': 'gbtree',\n",
       " 'colsample_bylevel': 0.45,\n",
       " 'colsample_bytree': 0.4,\n",
       " 'gamma': 0,\n",
       " 'learning_rate': 0.1,\n",
       " 'max_delta_step': 0,\n",
       " 'max_depth': 6,\n",
       " 'min_child_weight': 3,\n",
       " 'missing': None,\n",
       " 'n_estimators': 372,\n",
       " 'n_jobs': 6,\n",
       " 'nthread': None,\n",
       " 'objective': 'multi:softprob',\n",
       " 'random_state': 0,\n",
       " 'reg_alpha': 1.5,\n",
       " 'reg_lambda': 0.01,\n",
       " 'scale_pos_weight': 1,\n",
       " 'seed': 3,\n",
       " 'silent': True,\n",
       " 'subsample': 1}"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看最终调优模型参数\n",
    "xgb6.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=0.45,\n",
       "       colsample_bytree=0.4, gamma=0, learning_rate=0.1, max_delta_step=0,\n",
       "       max_depth=6, min_child_weight=3, missing=None, n_estimators=372,\n",
       "       n_jobs=6, nthread=None, objective='multi:softprob', random_state=0,\n",
       "       reg_alpha=1.5, reg_lambda=0.01, scale_pos_weight=1, seed=3,\n",
       "       silent=True, subsample=1)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 在训练集上训练模型\n",
    "xgb6.fit(X_train, y_train, eval_metric='mlogloss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 读取测试集数据\n",
    "dtest = pd.read_csv(dpath + 'RentListingInquries_FE_test.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_test = np.array(dtest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 对测试集进行预测\n",
    "y_test = xgb6.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 生成测试结果文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "result = pd.DataFrame(y_test, columns=['interest_level'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "result.to_csv(dpath + 'test_result.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Number of Occurences')"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEKCAYAAADaa8itAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAGpRJREFUeJzt3X3UXWV55/HvT14KWpC3yKKEGKSx\nDtWKkAEqXRZhigGtUKe0OrZEF6txFC3Wvogdx1Spra2tTukLHVZBYZYjAm0FEc1kEHS0giSoICAl\npQpZUMEGAbFAo9f8ce5HTsLJ8+wk++Rwku9nrbPO3te+9z7XOQ9wsfd973unqpAkqQ9Pm3QCkqTt\nh0VFktQbi4okqTcWFUlSbywqkqTeWFQkSb2xqEiSemNRkST1xqIiSerNzpNOYFvbb7/9auHChZNO\nQ5KmxurVq79dVfO6tN3hisrChQtZtWrVpNOQpKmR5Jtd24718leSvZJcluTrSW5L8tNJ9kmyMskd\n7X3v1jZJzkmyJslNSQ4fOs7S1v6OJEuH4kckubntc06SjPP7SJJmN+4+lT8DPl1VzwNeCNwGnAVc\nXVWLgKvbOsCJwKL2WgacC5BkH2A5cBRwJLB8phC1NsuG9lsy5u8jSZrF2IpKkj2BlwDnA1TV41X1\nHeBk4MLW7ELglLZ8MnBRDVwH7JXkAOBlwMqqWldVDwArgSVt255V9cUaTLV80dCxJEkTMM4zlecA\n9wMfSvLlJH+T5BnA/lV1L0B7f1ZrfyBw99D+a1tstvjaEfEnSbIsyaokq+6///6t/2aSpJHGWVR2\nBg4Hzq2qFwGP8MSlrlFG9YfUFsSfHKw6r6oWV9XiefM6DWCQJG2BcRaVtcDaqrq+rV/GoMh8q126\nor3fN9T+oKH95wP3zBGfPyIuSZqQsRWVqvoX4O4kP9FCxwO3AlcAMyO4lgKXt+UrgNPaKLCjgQfb\n5bEVwAlJ9m4d9CcAK9q2h5Mc3UZ9nTZ0LEnSBIz7PpW3AB9JsitwJ/B6BoXskiSnA3cBp7a2VwEn\nAWuA77W2VNW6JGcDN7R276mqdW35jcCHgd2BT7WXJGlCsqM9o37x4sXlzY+S1F2S1VW1uEvbHe6O\neklPfcf8+TGTTmG794W3fGEsx3VCSUlSbywqkqTeWFQkSb2xqEiSemNRkST1xqIiSeqNRUWS1BuL\niiSpNxYVSVJvLCqSpN5YVCRJvbGoSJJ6Y1GRJPXGoiJJ6o1FRZLUG4uKJKk3FhVJUm8sKpKk3lhU\nJEm9sahIknpjUZEk9caiIknqjUVFktQbi4okqTcWFUlSb8ZaVJJ8I8nNSb6SZFWL7ZNkZZI72vve\nLZ4k5yRZk+SmJIcPHWdpa39HkqVD8SPa8de0fTPO7yNJmt22OFN5aVUdVlWL2/pZwNVVtQi4uq0D\nnAgsaq9lwLkwKELAcuAo4Ehg+Uwham2WDe23ZPxfR5K0KZO4/HUycGFbvhA4ZSh+UQ1cB+yV5ADg\nZcDKqlpXVQ8AK4ElbdueVfXFqirgoqFjSZImYNxFpYD/k2R1kmUttn9V3QvQ3p/V4gcCdw/tu7bF\nZouvHRGXJE3IzmM+/jFVdU+SZwErk3x9lraj+kNqC+JPPvCgoC0DWLBgwewZS5K22FjPVKrqnvZ+\nH/D3DPpEvtUuXdHe72vN1wIHDe0+H7hnjvj8EfFReZxXVYuravG8efO29mtJkjZhbEUlyTOS7DGz\nDJwAfA24ApgZwbUUuLwtXwGc1kaBHQ082C6PrQBOSLJ366A/AVjRtj2c5Og26uu0oWNJkiZgnJe/\n9gf+vo3y3Rn431X16SQ3AJckOR24Czi1tb8KOAlYA3wPeD1AVa1LcjZwQ2v3nqpa15bfCHwY2B34\nVHtJkiZkbEWlqu4EXjgi/q/A8SPiBZyxiWNdAFwwIr4KeP5WJytJ6oV31EuSemNRkST1xqIiSeqN\nRUWS1BuLiiSpNxYVSVJvLCqSpN5YVCRJvZmzqCT54yR7JtklydVJvp3kV7ZFcpKk6dLlTOWEqnoI\neAWDSRyfC/z2WLOSJE2lLkVll/Z+EvDRoXm3JEnaQJe5vz7RnoPyb8CbkswDHh1vWpKkaTTnmUpV\nnQX8NLC4qv6dwQzCJ487MUnS9OnSUf90BrMHn9tCPwYsHmdSkqTp1KVP5UPA48CL2/pa4PfHlpEk\naWp1KSqHVNUfA/8OUFX/xujnw0uSdnBdisrjSXYHCiDJIcBjY81KkjSVuoz+Wg58GjgoyUeAY4DX\njTMpSdJ0mrOoVNXKJDcCRzO47HVmVX177JlJkqZOl9FfvwCsr6pPVtWVwPokp4w/NUnStOnSp7K8\nqh6cWamq7zC4JCZJ0ga6FJVRbbr0xUiSdjBdisqqJB9IckiS5yT5ILB63IlJkqZPl6LyFgY3P34M\nuJTBvF9njDMpSdJ06jL66xHgrG2QiyRpys1ZVJI8F/gtYOFw+6o6bnxpSZKmUZcO90uBvwb+Bvj+\neNORJE2zLn0q66vq3Kr6UlWtnnl1/YAkOyX5cpIr2/rBSa5PckeSjyXZtcV/pK2vadsXDh3jHS1+\ne5KXDcWXtNiaJF6ik6QJ61JUPpHkTUkOSLLPzGszPuNM4Lah9T8CPlhVi4AHgNNb/HTggar6ceCD\nrR1JDgVeDfwksAT4q1aodgL+EjgROBR4TWsrSZqQLkVlKYNn0v8Dg6HEq4FVXQ6eZD7wcgaXzkgS\n4DjgstbkQmDm7vyT2zpt+/Gt/cnAxVX1WFX9M7AGOLK91lTVnVX1OHAxPjxMkiaqy+ivg7fi+P8D\n+B1gj7a+L/Cdqlrf1tcCB7blA4G722euT/Jga38gcN3QMYf3uXuj+FGjkkiyDFgGsGDBgq34OpKk\n2XR68mOSdyY5r60vSvKKDvu9Arhvo/6XUc9hqTm2bW78ycGq86pqcVUtnjdv3ixZS5K2RpfRXx9i\ncMlr+MmPlwJXzrHfMcArk5wE7AbsyeDMZa8kO7ezlfnAPUPHPQhYm2Rn4JnAuqH4jOF9NhWXJE3A\n2J78WFXvqKr5VbWQQUf7Z6rqtcA1wC+2ZkuBy9vyFW2dtv0zVVUt/uo2OuxgYBHwJeAGYFEbTbZr\n+4wrOnwfSdKYdDlT6fvJj28HLk7y+8CXgfNb/HzgfyVZw+AM5dUAVXVLkkuAW4H1wBlV9f2Wy5uB\nFcBOwAVVdctW5CVJ2krb5MmPVXUtcG1bvpPByK2N2zwKnLqJ/d8LvHdE/Crgqs3JRZI0PrMWlTak\n9+vAq/DJj5KkOcxaVKqqkny8qo4APrmNcpIkTakuHfXXJfmPY89EkjT1uvSpvBR4Q5JvAo8wuARW\nVfVTY81MkjR1uhSVE8eehSRpu9ClqIy8S12SpI11KSqf5IlpUXYDDgZuZzBrsCRJP9RlQskXDK8n\nORx4w9gykiRNrS6jvzZQVTcCjgaTJD1Jl2fUv21o9WnA4cD9Y8tIkjS1uvSp7DG0vJ5BH8vfjicd\nSdI069Kn8u5tkYgkafp1eUjXyiR7Da3vnWTFeNOSJE2jLh3186rqOzMrVfUA8KzxpSRJmlZdisr3\nk/zwwe5Jno03REqSRujSUf/fgM8n+WxbfwmwbHwpSZKmVZeO+k+3Gx5nnqfyGz5PRZI0SpeO+l8A\n/r2qrqyqTwDrk5wy/tQkSdOmS5/K8qp6cGalddovH19KkqRp1aWojGrTpS9GkrSD6VJUViX5QJJD\nkjwnyQeB1eNOTJI0fboUlbcAjwMfAy4FHgXOGGdSkqTp1GX01yNJfh84u6oe2QY5SZKm1KxnKkne\nlOQu4JvAXUm+meRN2yY1SdK02WRRSfJO4BXAsVW1b1XtC7wUOLFtkyRpA7Odqfwq8KqqunMm0JZ/\nCTht3IlJkqbPrJe/qurREbF/A34w14GT7JbkS0m+muSWJO9u8YOTXJ/kjiQfS7Jri/9IW1/Tti8c\nOtY7Wvz2JC8bii9psTVJzur8rSVJYzFbUVmb5PiNg0mOA+7tcOzHgOOq6oXAYcCSJEcDfwR8sKoW\nAQ8Ap7f2pwMPVNWPAx9s7UhyKPBq4CeBJcBfJdkpyU7AXwInAocCr2ltJUkTMtvor18HLk/yeQb3\npRSDZ9MfA5w814GrqoDvttVd2quA44D/0uIXAr8HnNuO+XstfhnwF0nS4hdX1WPAPydZAxzZ2q2Z\nuTyX5OLW9ta5cpMkjccmz1Sq6hbg+cDngIXAc9ry89u2ObUziq8A9wErgX8CvlNV61uTtcCBbflA\n4O722euBB4F9h+Mb7bOpuCRpQma9T6X1qVywpQevqu8Dh7UnR/498B9GNWvv2cS2TcVHFcSRz3lJ\nsow2Xf+CBQtGNZEk9aDLHfVbrU1CeS2D6fP3SjJTzOYD97TltcBBAG37M4F1w/GN9tlUfNTnn1dV\ni6tq8bx58/r4SpKkEcZWVJLMm3m2fZLdgf8E3AZcA/xia7YUuLwtX9HWads/0/plrgBe3UaHHQws\nAr4E3AAsaqPJdmXQmX/FuL6PJGlus938eHV7/6MtPPYBwDVJbmJQAFZW1ZXA24G3tQ73fYHzW/vz\ngX1b/G3AWfDDvp1LGHTAfxo4o6q+3/pd3gysYFCsLuna1yNJGo/Z+lQOSPKzwCvbyKoN+jaq6sbZ\nDlxVNwEvGhG/kydGbw3HHwVO3cSx3gu8d0T8KuCq2fKQJG07sxWVdzE4W5gPfGCjbTNDgyVJ+qFN\nFpWqugy4LMl/r6qzt2FOkqQp1WXq+7OTvBJ4SQtd2/pGJEnawJyjv5L8IXAmg47yW4EzW0ySpA10\nedb8y4HDquoHAEkuBL4MvGOciUmSpk/X+1T2Glp+5jgSkSRNvy5nKn8IfDnJNQyGFb8Ez1IkSSN0\n6aj/aJJrGcxQHODtVfUv405MkjR9upypUFX34hQokqQ5bJMJJSVJOwaLiiSpN7MWlSRPS/K1bZWM\nJGm6zVpU2r0pX03ik60kSXPq0lF/AHBLki8Bj8wEq+qVY8tKkjSVuhSVd489C0nSdqHLfSqfTfJs\nYFFV/d8kTwd2Gn9qkqRp02VCyV8DLgP+ZwsdCHx8nElJkqZTlyHFZwDHAA8BVNUdwLPGmZQkaTp1\nKSqPVdXjMytJdmbw5EdJkjbQpah8NsnvArsn+TngUuAT401LkjSNuhSVs4D7gZuBNwBXAe8cZ1KS\npOnUZfTXD9qDua5ncNnr9qry8pck6UnmLCpJXg78NfBPDKa+PzjJG6rqU+NOTpI0Xbrc/PinwEur\nag1AkkOATwIWFUnSBrr0qdw3U1CaO4H7xpSPJGmKbfJMJcmr2uItSa4CLmHQp3IqcMM2yE2SNGVm\nO1P5+fbaDfgW8LPAsQxGgu0914GTHJTkmiS3JbklyZktvk+SlUnuaO97t3iSnJNkTZKbkhw+dKyl\nrf0dSZYOxY9IcnPb55wk2YLfQJLUk02eqVTV67fy2OuB36yqG5PsAaxOshJ4HXB1Vb0vyVkMhiy/\nHTgRWNReRwHnAkcl2QdYDixmcKa0OskVVfVAa7MMuI7BUOcl2NcjSRPTZfTXwcBbgIXD7eea+r49\n1/7etvxwktsYzBt2MoMzHoALgWsZFJWTgYvacOXrkuyV5IDWdmVVrWv5rASWJLkW2LOqvtjiFwGn\nYFGRpInpMvrr48D5DO6i/8GWfEiShcCLGNzrsn8rOFTVvUlm5hE7ELh7aLe1LTZbfO2IuCRpQroU\nlUer6pwt/YAkPwr8LfDWqnpolm6PURtqC+KjcljG4DIZCxb4EEtJGpcuQ4r/LMnyJD+d5PCZV5eD\nJ9mFQUH5SFX9XQt/q13Wor3PDE9eCxw0tPt84J454vNHxJ+kqs6rqsVVtXjevHldUpckbYEuReUF\nwK8B72NwI+SfAn8y105tJNb5wG1V9YGhTVcAMyO4lgKXD8VPa6PAjgYebJfJVgAnJNm7jRQ7AVjR\ntj2c5Oj2WacNHUuSNAFdLn/9AvCc4envOzoG+FXg5iRfabHfZVCcLklyOnAXg/teYDB66yRgDfA9\n4PUAVbUuydk8cW/Me2Y67YE3Ah8GdmfQQW8nvSRNUJei8lVgLzbzLvqq+jyj+z0Ajh/Rvhg8EGzU\nsS4ALhgRXwU8f3PykiSNT5eisj/w9SQ3AI/NBOcaUixJ2vF0KSrLx56FJGm70OV5Kp/dFolIkqZf\nlzvqH+aJ+z92BXYBHqmqPceZmCRp+nQ5U9ljeD3JKcCRY8tIkjS1utynsoGq+jhw3BhykSRNuS6X\nv141tPo0npgtWJKkDXQZ/fXzQ8vrgW8wmFFYkqQNdOlT2drnqkiSdhCzPU74XbPsV1V19hjykSRN\nsdnOVB4ZEXsGcDqwL2BRkSRtYLbHCf/pzHJ7HPCZDCZ5vJjBTMWSJG1g1j6V9nz4twGvZfDo38Pb\ns+ElSXqS2fpU3g+8CjgPeEFVfXebZSVJmkqz3fz4m8CPAe8E7knyUHs9nOShbZOeJGmazNanstl3\n20uSdmwWDklSbywqkqTedJmmRZo6d73nBZNOYYew4F03TzoFPcV4piJJ6o1FRZLUG4uKJKk3FhVJ\nUm8sKpKk3lhUJEm9sahIknpjUZEk9WZsRSXJBUnuS/K1odg+SVYmuaO9793iSXJOkjVJbkpy+NA+\nS1v7O5IsHYofkeTmts85STKu7yJJ6macZyofBpZsFDsLuLqqFgFXt3WAE4FF7bUMOBd++DyX5cBR\nwJHA8plC1NosG9pv48+SJG1jYysqVfU5YN1G4ZMZPOyL9n7KUPyiGrgO2CvJAcDLgJVVta49HGwl\nsKRt27OqvlhVBVw0dCxJ0oRs6z6V/avqXoD2/qwWPxC4e6jd2habLb52RHykJMuSrEqy6v7779/q\nLyFJGu2p0lE/qj+ktiA+UlWdV1WLq2rxvHnztjBFSdJctnVR+Va7dEV7v6/F1wIHDbWbD9wzR3z+\niLgkaYK2dVG5ApgZwbUUuHwoflobBXY08GC7PLYCOCHJ3q2D/gRgRdv2cJKj26iv04aOJUmakLE9\nTyXJR4Fjgf2SrGUwiut9wCVJTgfuAk5tza8CTgLWAN8DXg9QVeuSnA3c0Nq9p6pmOv/fyGCE2e7A\np9pLkjRBYysqVfWaTWw6fkTbAs7YxHEuAC4YEV8FPH9rcpQk9eup0lEvSdoOWFQkSb2xqEiSemNR\nkST1xqIiSeqNRUWS1BuLiiSpNxYVSVJvLCqSpN5YVCRJvbGoSJJ6Y1GRJPXGoiJJ6o1FRZLUG4uK\nJKk3FhVJUm8sKpKk3lhUJEm9sahIknpjUZEk9WbnSSfwVHbEb1806RS2e6vff9qkU5DUI89UJEm9\nsahIknpjUZEk9caiIknqjUVFktSbqS8qSZYkuT3JmiRnTTofSdqRTXVRSbIT8JfAicChwGuSHDrZ\nrCRpxzXVRQU4ElhTVXdW1ePAxcDJE85JknZY015UDgTuHlpf22KSpAmY9jvqMyJWT2qULAOWtdXv\nJrl9rFlNzn7AtyedxObInyyddApPJVP392P5qH8Fd1hT9ffLr2/W3+7ZXRtOe1FZCxw0tD4fuGfj\nRlV1HnDetkpqUpKsqqrFk85DW8a/33Tz7zcw7Ze/bgAWJTk4ya7Aq4ErJpyTJO2wpvpMparWJ3kz\nsALYCbigqm6ZcFqStMOa6qICUFVXAVdNOo+niO3+Et92zr/fdPPvB6TqSf3akiRtkWnvU5EkPYVY\nVLYTTlczvZJckOS+JF+bdC7aPEkOSnJNktuS3JLkzEnnNGle/toOtOlq/hH4OQbDrG8AXlNVt040\nMXWS5CXAd4GLqur5k85H3SU5ADigqm5MsgewGjhlR/53zzOV7YPT1UyxqvocsG7SeWjzVdW9VXVj\nW34YuI0dfFYPi8r2welqpAlLshB4EXD9ZDOZLIvK9qHTdDWSxiPJjwJ/C7y1qh6adD6TZFHZPnSa\nrkZS/5LswqCgfKSq/m7S+UyaRWX74HQ10gQkCXA+cFtVfWDS+TwVWFS2A1W1HpiZruY24BKnq5ke\nST4KfBH4iSRrk5w+6ZzU2THArwLHJflKe5006aQmySHFkqTeeKYiSeqNRUWS1BuLiiSpNxYVSVJv\nLCqSpN5YVDTVkvxDhzZvTfL0Medx2KaGkiY5NsmVPX9e78dsx93i3yrJKUkO7TsnTReLiqZaVb24\nQ7O3Apv1H8o28/PmOAzYHu5P2OzfasgpgEVlB2dR0VRL8t32fmySa5NcluTrST6SgV8Hfgy4Jsk1\nre0JSb6Y5MYkl7Z5m0jyjSTvSvJ54NQkhyT5dJLVSf5fkue1dqcm+VqSryb5XJvF4D3AL7eb3355\nlnyf0Z6fckOSLyc5ucWvT/KTQ+2uTXLEptrPcvzdknwoyc2t/Utb/HVJ/q59nzuS/PGIfTfnt3pf\nkluT3JTkT5K8GHgl8P72GxzS6Q+o7U9V+fI1tS/gu+39WOBBBvOePY3BHeo/07Z9A9ivLe8HfA54\nRlt/O/CuoXa/M3Tsq4FFbfko4DNt+WbgwLa8V3t/HfAXm8jxWODKtvwHwK/M7MvgOTjPAH4DeHeL\nHwD84xztf3jMjT7rN4EPteXnAXcBu7X87gSe2da/CRw0Yv85fytgH+B2nrh5euY3+DDwi5P+Z8LX\nZF87z15ypKnypapaC5DkK8BC4PMbtTmawSWaLwymbWJXBgVoxsfa/j8KvBi4tLUD+JH2/gXgw0ku\nATZ3AsETgFcm+a22vhuwALgEWAksB34JuHSO9pvyM8CfA1TV15N8E3hu23Z1VT3Yvt+twLPZ8JEJ\nG9vUb/UQ8CjwN0k+CfTet6PpZVHR9uSxoeXvM/qf7wArq+o1mzjGI+39acB3quqwjRtU1X9NchTw\ncuArSZ7UZhYB/nNV3f6kDcm/Jvkp4JeBN8zWPsn+sxx/U7r8Phsfa+RvleRI4HgGk5e+GThujmNp\nB2GfinYEDwN7tOXrgGOS/DhAkqcnee7GO9TgmRj/nOTU1i5JXtiWD6mq66vqXcC3GTx2YPgzZrMC\neEub3ZYkLxradjHwO8Azq+rmDu1H+Rzw2tb2uQzOap5UwGYx52/VzuKeWVVXMejYP2zEvtpBWVS0\nIzgP+FSSa6rqfgb9Cx9NchOD/3A+bxP7vRY4PclXgVt44hHN728d4V9j8B/xrwLXAIfO1VEPnA3s\nAtzU9j97aNtlDP7P/5KO7Uf5K2CnJDczuJT3uqp6bI59hnX5rfYArmyxzzLoD4JBUfztNkDAjvod\nlLMUS5J645mKJKk3FhVJUm8sKpKk3lhUJEm9sahIknpjUZEk9caiIknqjUVFktSb/w+GHJpnPGrr\n4wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1118a3048>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 查看预测结果分布\n",
    "sns.countplot(y_test)\n",
    "plt.xlabel('interest level on test')\n",
    "plt.ylabel('Number of Occurences')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "预测结果中, interest_level 为 0 和 1 的分类样本比例还是偏低."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
